Anand Shah

PERL script to test SMTP connections

2009-09-25T14:39:00.001+05:30

SMTP CHECK PERL SCRIPT:
USAGE:- ./perlscript smtp servername

#!/usr/bin/perl

use Net::SMTP;

$filename = $ARGV[0];
$smtpserver = $ARGV[1];
open (FH, "<$filename") || die "Could not open email file $!\n";
(@lines) = ; # read file into list
close(FH);

#open(EFILE2,"<$text_attach") || die "Could not open email file $!\n";
#(@lines) = ; # read file into list
#close(EFILE2);
#close(FH);

#my $ServerName = $smtpserver;
#$smtp = Net::SMTP->new($ServerName, Debug => 0);
#$smtp = Net::SMTP->new("xx.xx.xx");

$smtp = Net::SMTP->new($smtpserver,
Hello => 'mail.testmail.co.in',
Port => 25,
Timeout => 30,
Debug => 1,
);

$from="test\@rediffmail.com";
$cc_list="test\@yahoo.com";
#$recip2="f\@lehman.com";
$recip2="test\@anilkumar.com";

$smtp->mail("$from");

###############################################
##$smtp->to("$recip,$cc_list");
##$smtp->cc("$cc_list");
##$smtp->bcc("$bcc");
###############################################

$smtp->recipient($recip2,$cc_list);
$smtp->data;

$smtp->datasend("To: $recip2\n");
$smtp->datasend("From: $from\n");

$smtp->datasend("Cc: $cc_list\n");
$smtp->datasend("Bcc: $bcc\n");
$smtp->datasend("Subject: $subject\n");

$smtp->datasend("MIME-Version: 1.0 \n");
$smtp->datasend("Content-Type: text/html; charset=us-ascii \n");
$smtp->datasend("\n");

$smtp->datasend(@lines);

$smtp->dataend();
$smtp->quit;

Perl script to uninstall Modules Cleanly

2009-09-25T14:37:00.001+05:30

This script detects the installed perl modules:-

#!/usr/local/bin/perl -w

use strict;
use IO::Dir;
use ExtUtils::Packlist;
use ExtUtils::Installed;

sub emptydir($) {
my ($dir) = @_;
my $dh = IO::Dir->new($dir) || return(0);
my @count = $dh->read();
$dh->close();
return(@count == 2 ? 1 : 0);
}

# Find all the installed packages
print("Finding all installed modules...\n");
my $installed = ExtUtils::Installed->new();

foreach my $module (grep(!/^Perl$/, $installed->modules())) {
my $version = $installed->version($module) || "???";
print("Found module $module Version $version\n");
print("Do you want to delete $module? [n] ");
my $r = ; chomp($r);
if ($r && $r =~ /^y/i) {
# Remove all the files
foreach my $file (sort($installed->files($module))) {
print("rm $file\n");
unlink($file);
}
my $pf = $installed->packlist($module)->packlist_file();
print("rm $pf\n");
unlink($pf);
foreach my $dir (sort($installed->directory_tree($module))) {
if (emptydir($dir)) {
print("rmdir $dir\n");
rmdir($dir);
}
}
}
}

Installing Apache Tomcat on Linux

2009-07-03T15:03:00.001+05:30

Installing Tomcat on Linux - Tomcat Installation on Linux - Apache Tomcat HOWTO

This article is a step by step guide for installing Apache Tomcat 6.0 (6.0.18) on 64-bit Debian Linux 4.0.
It covers the setup of multiple Tomcat JVM instances on a single Linux server.
The instructions in this guide are applicable to most other Linux distributions.

Contents

* Introduction

* Installing Java Runtime Environment

* Installing Tomcat Software

    Starting/Stopping Tomcat

    Switching to Tomcat User Account

* Setting Up First Tomcat JVM Instance

    Setting up Directories and Files

    Configuring Tomcat Network Ports

    Starting First Tomcat Instance

    Relaying HTTP Port 80 Connections to Tomcat Port 8080

    Connecting to First Tomcat Instance Using Default HTTP Port

* Setting Up a Web Application for First Tomcat JVM Instance

    Setting up Web Application Layout

    Configuring Web Application

    Home Page for Web Application

    Restarting First Tomcat Instance

* Deploying Java Servlet for Web Application in First Tomcat JVM Instance

    Setting up Java Servlet Layout

    JAR Files

    Creating a Java Servlet

    Configuring the Java Servlet

    Testing and Executing the Java Servlet

* Setting Up Second Tomcat JVM Instance

    General

    Steps for Second Tomcat JVM Instance and Application

* Bibliography and References

Introduction

This article discusses how to install Apache Tomcat 6.0 (6.0.18) on 64-bit Debian Linux 4.0.
Additionally it shows how to setup multiple Tomcat JVM instances on a single Linux server.
For each Tomcat JVM instance a web application and Java servlet example is configured.
The Tomcat installation steps outlined in this article are also applicable to most other Linux distributions.

Note that this document comes without warranty of any kind. But every effort has been made to provide the information as accurate as possible.
I welcome emails from any readers with comments, suggestions, and corrections at webmaster_at_puschitz.com.

Installing Java Runtime Environment

To run Tomcat, you need Java Standard Edition (Java SE), also known as the JDK.

For the Tomcat installation I used SUN's latest Java SE JDK that was available at the time of this writing:
Java SE Development Kit (JDK) 6 Update 10 (6u10).
Regarding Java SE 6, Platform Name and Version Numbers, see
http://java.sun.com/javase/6/webnotes/version-6.html.
And for the whole Java version history I recommend the Wiki article
http://en.wikipedia.org/wiki/Java_version_history.

You can download SUN's latest Java JDKs at:
http://java.sun.com/javase/downloads/index.jsp.

For my 64-bit Debian system I selected the 64-bit JDK multiplatform binary for Linux: jdk-6u10-linux-x64.bin.

I downloaded the binary file to /tmp and installed it as follows as root:

# mkdir -p /usr/java 

# cd /usr/java 

#

# chmod 700 /tmp/jdk-6u10-linux-x64.bin 

# /tmp/jdk-6u10-linux-x64.bin 

...

   creating: jdk1.6.0_10/

   creating: jdk1.6.0_10/db/

   creating: jdk1.6.0_10/db/bin/

  inflating: jdk1.6.0_10/db/bin/ij   

  inflating: jdk1.6.0_10/db/bin/NetworkServerControl  

  inflating: jdk1.6.0_10/db/bin/setNetworkClientCP.bat  

  inflating: jdk1.6.0_10/db/bin/derby_common.sh  

...

Done.

# export JAVA_HOME=/usr/java/jdk1.6.0_10 

# export PATH=$JAVA_HOME/bin:$PATH 

#

# which java 

/usr/java/jdk1.6.0_10/bin/java

# java -version 

java version "1.6.0_10"

Java(TM) SE Runtime Environment (build 1.6.0_10-b33)

Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)

#

Installing Tomcat Software

Download the latest Tomcat 6.x version from
http://tomcat.apache.org/download-60.cgi.
For Debian I downloaded the Binary Core Distribution file apache-tomcat-6.0.18.tar.gz which was the
latest version at the time of this writing.

Once you downloaded the tar file make sure the MD5 checksum matches the value posted on Tomcat's web site, see
http://www.apache.org/dist/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.tar.gz.md5:

# md5sum /tmp/apache-tomcat-6.0.18.tar.gz 

8354e156f097158f8d7b699078fd39c1  /tmp/apache-tomcat-6.0.18.tar.gz

Installing Tomcat from a binary release (tar file) requires manual creation of the Tomcat user account.
This is not necessary if you install the Tomcat RPM package on a Linux system that supports RPMs.

For security reasons I created a user account with no login shell for running the Tomcat server:

# groupadd tomcat 

# useradd -g tomcat -s /usr/sbin/nologin -m -d /home/tomcat tomcat

(It should be noted that other Linux systems have nologin under /sbin not /usr/sbin)

Next I extracted the tar file to /var/lib and changed the ownership of all files and directories to tomcat:

# cd /var/lib 

# tar zxvf /tmp/apache-tomcat-6.0.18.tar.gz 

# chown -R tomcat.tomcat /var/lib/apache-tomcat-6.0.18

The get the Tomcat version of the newly installed Tomcat, run:

# /var/lib/apache-tomcat-6.0.18/bin/version.sh 

Using CATALINA_BASE:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_HOME:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_TMPDIR: /var/lib/apache-tomcat-6.0.18/temp

Using JRE_HOME:       /usr

Server version: Apache Tomcat/6.0.18

Server built:   Jul 22 2008 02:00:36

Server number:  6.0.18.0

OS Name:        Linux

OS Version:     2.6.18-6-amd64

Architecture:   x86_64

JVM Version:    1.4.2

JVM Vendor:     Free Software Foundation, Inc.

#

Starting/Stopping Tomcat

Now try to startup the Tomcat server to see whether the default Tomcat home page is being displayed.

For security reasons I don't run the Tomcat server as user root but as tomcat which was created with no login shell.
Therefore, to run Tomcat use the su command with the -p option to preserves
all the environment variables when switching to tomcat (more on the Tomcat environment variables later).
And since the tomcat account has no login shell, it needs to be specified with the -s option.
(You may want to use this su command if you plan on writing and implementing a system startup and shutdown script for system reboots.)

# export JAVA_HOME=/usr/java/jdk1.6.0_10 

# export PATH=$JAVA_HOME/bin:$PATH 

# export CATALINA_HOME=/var/lib/apache-tomcat-6.0.18 

# export CATALINA_BASE=/var/lib/apache-tomcat-6.0.18 

#

 

# su -p -s /bin/sh tomcat $CATALINA_HOME/bin/startup.sh 

Using CATALINA_BASE:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_HOME:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_TMPDIR: /var/lib/apache-tomcat-6.0.18/temp

Using JRE_HOME:       /usr/java/jdk1.6.0_10

#

Now verify that Tomcat was started successfully by opening the URL http://localhost:8080 (Port number 8080 is the default port used by Tomcat).
Note that you should also be able to use the name of your server instead of localhost.
Once you opened the URL in your browser you should see Tomcat's Congratulation page.
If you don't see the page, check the log files under $CATALINA_HOME/logs (/var/lib/apache-tomcat-6.0.18/logs).

Before you continue with the next steps, make sure to shut down Tomcat since we want to run the Tomcat server out of a separate
application directory which is covered in the next chapter.

# su -p -s /bin/sh tomcat $CATALINA_HOME/bin/shutdown.sh 

Using CATALINA_BASE:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_HOME:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_TMPDIR: /var/lib/apache-tomcat-6.0.18/temp

Using JRE_HOME:       /usr/java/jdk1.6.0_10

#

Switching to Tomcat User Account

Most of the next steps in this article assume that you switched to the tomcat user account.
If you see a '$' prompt, then the steps in this article are executed as the tomcat user.
If you see a '#' prompt, then the steps are executed as root.

Since for security reasons the tomcat user has no login shell, it needs to be specified with the -s option when switching from
root to tomcat:

# su - -s /bin/sh tomcat 

$ id 

uid=1001(tomcat) gid=1001(tomcat) groups=1001(tomcat)

$

Note that non-root users cannot switch to the tomcat account.

Setting Up First Tomcat JVM Instance

It is recommended not to store the web applications's files in Tomcat's distribution directory tree.
For example, having a separate directory makes Tomcat upgrades easier since it won't
overwrite configuration files like server.xml. And since this tutorial shows how to run two
Tomcat instances concurrently on a single Linux server, two separate directories are needed anyway.
It should be noted here that it's also possible to run multiple web applications per Tomcat JVM instance.
This HOWTO shows the creation and configuration of one web application for each Tomcat instance.

Setting up Directories and Files

In the following example I setup the first Tomcat JVM instance under the base directory /opt/tomcat-instance/sales.example.com.
It's a good practice to name the base directory after the site name, in this example sales.example.com.

Creating a new base directory for a new instance requires the creation and copying of various directories and
configuration files. Execute the following commands as root:

# mkdir -p /opt/tomcat-instance/sales.example.com 

# cd /opt/tomcat-instance/sales.example.com 

#

# cp -a /var/lib/apache-tomcat-6.0.18/conf . 

# mkdir common logs temp server shared webapps work 

#

# chown -R tomcat.tomcat /opt/tomcat-instance

Most of the remaining steps are executed as the tomcat user. So make sure you switch from root
to tomcat:

# su - -s /bin/sh tomcat 

$ id 

uid=1001(tomcat) gid=1001(tomcat) groups=1001(tomcat)

$

Next I created an environment file for the new Tomcat instance.
This will be useful for easily setting the environment variables when starting/stopping the new Tomcat instance:

$ cat > /opt/tomcat-instance/sales.env << EOF

export JAVA_HOME=/usr/java/jdk1.6.0_10

export PATH=\$JAVA_HOME/bin:\$PATH

export CATALINA_HOME=/var/lib/apache-tomcat-6.0.18 

export CATALINA_BASE=/opt/tomcat-instance/sales.example.com 

EOF 

$

$ cat /opt/tomcat-instance/sales.env 

export JAVA_HOME=/usr/java/jdk1.6.0_10

export PATH=$JAVA_HOME/bin:$PATH

export CATALINA_HOME=/var/lib/apache-tomcat-6.0.18

export CATALINA_BASE=/opt/tomcat-instance/sales.example.com

$

CATALINA_HOME is the base directory of Tomcat that contains all the libraries, scripts etc. for Tomcat.
This is the parent directory of the extracted Tomcat tar file.

CATALINA_BASE is the base directory of the new Tomcat instance, which in this example points to
/opt/tomcat-instance/sales.example.com.

Configuring Tomcat Network Ports

Since this is the first Tomcat instance that's being created here, the default port numbers can be left unchanged
in $CATALINA_BASE/conf/server.xml (/opt/tomcat-instance/sales.example.com/conf/server.xml):

    <Server port="8005" shutdown="SHUTDOWN">

 

    <Connector port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

 

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

However, these port numbers will have to be changed for the second Tomcat instance though, see
Steps for Second Tomcat JVM Instance and Application.

Starting First Tomcat Instance

To start the newly created Tomcat JVM instance, ensure that the environment variables are set for the new instance and execute the startup script:

$ source /opt/tomcat-instance/sales.env 

$ $CATALINA_HOME/bin/startup.sh 

Using CATALINA_BASE:   /opt/tomcat-instance/sales.example.com

Using CATALINA_HOME:   /var/lib/apache-tomcat-6.0.18

Using CATALINA_TMPDIR: /opt/tomcat-instance/sales.example.com/temp

Using JRE_HOME:       /usr/java/jdk1.6.0_10

$

If everything has been configured correctly, you should now see an empty white page when opening the URL http://localhost:8080.
Note that instead of localhost you should also be able to use the name of your server.

If you get an error in the browser instead of an empty page, check the log files under
$CATALINA_BASE/logs (/opt/tomcat-instance/sales.example.com/logs).
Note that since CATALINA_BASE has been changed for the new Tomcat instance, the logs are no longer written to /var/lib/apache-tomcat-6.0.18/logs.

Relaying HTTP Port 80 Connections to Tomcat Port 8080

By default, Tomcat listens on port 8080. To have the Tomcat server itself listen on HTTP port 80, Tomcat would have to run as root
since only root can listen on ports below 1024 on Linux. But for security reasons this is not recommended.
The solution I prefer is to relay port 80 TCP connections to port 8080 using the Netfilter package
that comes with Linux.
An alternate solution would be to use a service wrapper like jsvc from the
Jakarta Commons Daemon project. But this solution would require the installation and maintenance of another
piece of software on my system that I want to avoid.

The Netfilter package that comes already with Linux is transparent to Tomcat. The following steps show how to relay port 80 TCP connections to Tomcat's
port 8080 using the iptables command from the Netfilter package. Note that these steps must be executed as root:

# iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080 

# iptables -t nat -I OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 8080

The first rule redirects incoming requests on port 80 generated from other computer nodes, and the second rule
redirects incoming requests on port 80 generated from the local node where Tomcat is running.

To see the newly configured rules, run:

# iptables -t nat -L 

Chain PREROUTING (policy ACCEPT)

target     prot opt source               destination         

REDIRECT   tcp  --  anywhere             anywhere            tcp dpt:www redir ports 8080 

 

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination         

 

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination         

REDIRECT   tcp  --  anywhere             anywhere            tcp dpt:www redir ports 8080 

#

To remove the NAT rules we just created, you can run the iptables -t nat -F command which flushes and deletes the rules.
Note that this will also flush any other rules that may have been configured on your system!
For more information on iptables, see
netfilter/iptables documentation.

To make the rules permanent for reboots, you can use the following option outlined here for Debian
(other Linux distributions have other methods). First save the newly created rules in a file:

# iptables-save > /etc/iptables.conf

Then edit the /etc/network/interfaces file and add the line highlighted in blue for the public network interface. For example:

iface eth0 inet static

        address 192.168.1.23

        netmask 255.255.255.0

        network 192.168.1.0

        broadcast 192.168.1.255

        gateway 192.168.1.1

        pre-up iptables-restore < /etc/iptables.conf

The pre-up configuration in this example activates the iptables rules on my system before the public interface eth0 comes up.
So the rules can be seen with iptables -t nat -L after each reboot.
Note that for security reasons it's important that firewall rules are established before the network interfaces come up.
Even though this is not an issue for relaying Tomcat connections, as a matter of good practice, the iptables rules should always
be established before the network comes up.

It should be noted here that there is one Tomcat configuration parameter that you may or may not want to change, the proxyPort
parameter in the server.xml file.
Since Tomcat still receives requests on port 8080 as they are relayed by the Linux Netfilter system from port 80,
Tomcat may display port 8080 in the URL depending on the application's content.
So if you want to change it to port 80, the proxyPort parameter would need to be added in the $CATALINA_BASE/conf/server.xml (/opt/tomcat-instance/sales.example.com/conf/server.xml).
file for port 8080:

 

    <Connector port="8080" protocol="HTTP/1.1" proxyPort="80" 

               connectionTimeout="20000"

               redirectPort="8443" />

After that you need to restart Tomcat to make this change effective.

Connecting to First Tomcat Instance Using Default HTTP Port

If iptables have been configured correctly, you should now be able to open the URL
http://localhost and see an empty white page.
You could also use the URL http://localhost:80 (port 80 is the default port used by browsers) or the name of your server.
If you get an error in the browser instead of an empty page, check the iptables configuration and check the log files under
$CATALINA_BASE/logs (/opt/tomcat-instance/sales.example.com/logs).
Note that since CATALINA_BASE was changed for the new Tomcat instance, the logs are no longer written to /var/lib/apache-tomcat-6.0.18/logs.

Setting Up a Web Application for First Tomcat JVM Instance

You can setup multiple web applications for each Tomcat JVM instance. In this guide we are setting up one web application for
each Tomcat JVM instance.

First make sure to switch to the tomcat user account and source in the environment variables for the remaining steps:

# su - -s /bin/sh tomcat 

$ source /opt/tomcat-instance/sales.env

Setting up Web Application Layout

In the previous chapter the first Tomcat JVM instance was setup under the base directory $CATALINA_BASE (/opt/tomcat-instance/sales.example.com).
In the following example I create a new directory called "sales" under $CATALINA_BASE/webapps which
will become the root directory for the first web application, that is $CATALINA_BASE/webapps/sales. In Tomcat web application root directories
are created under $CATALINA_BASE/webapps by default.

$ mkdir $CATALINA_BASE/webapps/sales

Configuring Web Application

To configure Tomcat to recognize the new web application under $CATALINA_BASE/webapps/sales (/opt/tomcat-instance/sales.example.com/webapps/sales),
the $CATALINA_BASE/conf/server.xml file needs to be edited.
This is done by adding a new Context element with the path and docBase attributes.
Note that Tomcat refers to webapps as "context". So Context here represents the configuration of a web application.
The path attribute is the application name used within the URL, and the docBase attribute is the absolute path name
of the new web application root under $CATALINA_BASE/webapps:

 

      <Host name="localhost"  appBase="webapps"

            unpackWARs="true" autoDeploy="true"

            xmlValidation="false" xmlNamespaceAware="false">

 

        <Context docBase="sales" path="/mysales"/>

In this example you can see that appBase already points to webapps by default, that is $CATALINA_BASE/webapps.
The newly added path attribute points to
the sales directory under $CATALINA_BASE/webapps which is the location for the new application.
And the docBase attribute is set to mysales
which stands for the application name within the URL, i.e. "http://localhost/mysales" or "http://localhost:8080/mysales".
Make sure to add this new Context element inside the Host container element for 'localhost' which is the default host name.

Home Page for Web Application

To have a starting page for the new web application, you can simply create and add a index.html file under the web application's root directory
$CATALINA_BASE/webapps/sales (/opt/tomcat-instance/sales.example.com/webapps/sales).
You could also create your own JSP page here.
For testing purposes here is a simple index.html example for the new application:

$ cat > $CATALINA_BASE/webapps/sales/index.html << EOF

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

"http://www.w3.org/TR/html4/loose.dtd">

<HTML>

<HEAD><META http-equiv=Content-Type content="text/html"></HEAD>

<BODY>

<H3>Apache Tomcat Sales Home Page</H3>

</BODY>

</HTML>

EOF 

$

Restarting First Tomcat Instance

Now check whether the new web application has been configured correctly.
To do that, run the following commands to restart the new Tomcat JVM instance:

$ source /opt/tomcat-instance/sales.env 

$ $CATALINA_HOME/bin/shutdown.sh 

$ $CATALINA_HOME/bin/startup.sh

If everything was configured correctly, you should now see the default home page for the new web application when
opening the URL http://localhost/mysales or http://localhost/mysales:8080.
Instead of localhost you should also be able to use the name of your server.
If you get the error 'java.net.ConnectException: Connection refused' when you shutdown Tomcat, then Tomcat
was probably not running.
If you don't see the home page, check the log files under $CATALINA_BASE/logs.

Deploying Java Servlet for Web Application in First Tomcat JVM Instance

Setting up Java Servlet Layout

To follow the Java Servlet Specification for the new "sales" web application, I created the class directory for the Java class files
under the new directory $CATALINA_BASE/webapps/sales/WEB-INF, see also
Packaging Web Components.
The WEB-INF directory is protected from access by browsers, meaning they are unbrowsable and safe from client views.

The classes directory under WEB-INF is where web components and server-side utility classes should go.
To create the WEB-INF and classes directories, run the following command:

$ mkdir -p $CATALINA_BASE/webapps/sales/WEB-INF/classes

JAR Files

Most Java servlets also need JAR (Java ARchive) files which should be put under the lib directory.
Since it's a good practice to keep the application separate from the Tomcat distribution directory tree, I created
a new lib directory under $CATALINA_BASE/webapps/sales/WEB-INF which is consistent with
WAR's hierarchical directory structure.

$ mkdir $CATALINA_BASE/webapps/sales/WEB-INF/lib

The Java servlet example below requires the servlet-api.jar JAR file. This JAR is already available in the Tomcat distribution directory tree $CATALINA_HOME/lib.
You could copy this JAR file to the application's new lib directory $CATALINA_BASE/webapps/sales/WEB-INF/lib, but then you would get the following
warning in the $CATALINA_BASE/logs/catalina.out log file when you startup Tomcat:

INFO: validateJarFile(/opt/tomcat-instance/sales.example.com/webapps/sales/WEB-INF/lib/servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class

Tomcat shows this warning since it tries now to load the JAR file twice, first from $CATALINA_HOME/lib and then from $CATALINA_BASE/webapps/sales/WEB-INF/lib.
Even though it's not going to be a problem for Tomcat, it's better not to keep JARs in two places. Since the servlet-api.jar JAR file already exists
in the Tomcat distribution directory, I did not copy it to the $CATALINA_BASE/webapps/sales/WEB-INF/lib directory.
I use this directory for application specific JARs that don't come with the Tomcat distribution.
You could also remove the JAR in $CATALINA_HOME/lib but remember that it will reappier the next time you upgrade the Tomcat software.

Creating a Java Servlet

Since server-side classes are supposed to go to the WEB-INF/classes directory, I created the following class file example under
$CATALINA_BASE/webapps/sales/WEB-INF/classes (/opt/tomcat-instance/sales.example.com/webapps/sales/WEB-INF/classes) and saved it
as Sales.java:

$ cat $CATALINA_BASE/webapps/sales/WEB-INF/classes/Sales.java 

import java.io.*;

import javax.servlet.*;

import javax.servlet.http.*;

 

public class Sales extends HttpServlet {

 

    public void doGet(HttpServletRequest request, HttpServletResponse response)

    throws IOException, ServletException

    {

        response.setContentType("text/html");

        PrintWriter out = response.getWriter();

        out.println("<html>");

        out.println("<head>");

        out.println("<title>Sales Page</title>");

        out.println("</head>");

        out.println("<body>");

        out.println("<h1>Executing Sales ...</h1>");

        out.println("</body>");

        out.println("</html>");

    }

}

To compile the new Java servlet, the servlet-api.jar JAR file is needed which can be specified with either
the -classpath option or the CLASSPATH environment variable. The -classpath option for SDK tools is preferred
over the CLASSPATH environment variable since it can be set individually for each application without affecting others.
In the following example I specify the path of the class directory with the basename '*'
(if you are unfamiliar with basename, see 'man basename').
This is equivalent to specifying all files with the extensions .jar or .JAR files in the directory and therefore individual JAR files like
servlet-api.jar don't need to be specified.

The following command should now compile the Java servlet without errors:

$ cd $CATALINA_BASE/webapps/sales/WEB-INF/classes 

$ javac -classpath "$CATALINA_HOME/lib/*" Sales.java 

$ ls 

Sales.class  Sales.java

$

Configuring the Java Servlet

To configure servlets and other components for an application, an XML file called web.xml needs to be configured.
The format of this file is defined in the Java Servlet Specification. In Tomcat, this file exists in two place:

  $CATALINA_BASE/conf/web.xml

  $CATALINA_BASE/webapps/{your-appname}/WEB-INF/web.xml

The first one is the default web.xml file which is the base for all web applications in a Tomcat JVM instance,
and the latter one is for the web application where WEB-INF resides for overwriting application specific settings.

For the newly created Java servlet "Sales" I created a new web.xml file under $CATALINA_BASE/webapps/sales/WEB-INF:

$ cat $CATALINA_BASE/webapps/sales/WEB-INF/web.xml 

<?xml version="1.0" encoding="ISO-8859-1"?>

 

<web-app xmlns="http://java.sun.com/xml/ns/javaee"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"

    version="2.5">

 

  <servlet>

    <servlet-name>servlet_sales</servlet-name>

    <servlet-class>Sales</servlet-class>

  </servlet>

 

  <servlet-mapping>

    <servlet-name>servlet_sales</servlet-name>

    <url-pattern>/execute</url-pattern>

  </servlet-mapping>

 

</web-app>

For each servlet there is a <servlet> element.
It identifies the servlet name (<servlet-name>) and the Java class name (<servlet-class>).
The servlet mapping (<servlet-mapping>) maps a URI to the servlet name (<servlet-name>).
In the above example "/execute" in "http://localhost:8080/mysales/execute" maps to "servlet_sales" which points to the "Sales" servlet class.
Note that the order of these elements is important.
So when you open the URL "http://localhost:8080/mysales/execute", the "Sales" Java servlet will be executed.

In the following example I updated the $CATALINA_BASE/webapps/sales/index.html file to provide an
entry point to the new Java servlet:

$ cat $CATALINA_BASE/webapps/sales/index.html 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

"http://www.w3.org/TR/html4/loose.dtd">

<HTML>

<HEAD><META http-equiv=Content-Type content="text/html"></HEAD>

<BODY>

<H3>Apache Tomcat Sales Home Page</H3>

<a href="/mysales/execute">Execute Sales</a> 

</BODY>

</HTML>

$

Testing and Executing the Java Servlet

Note that if you run javac with the -classpath option or the CLASSPATH environment variable in the same shell before you startup Tomcat,
you will get java.lang.NoClassDefFoundError / java.lang.ClassNotFoundException
errors in your browser when you execute a servlet. To avoid this, simply re-login as the tomcat user before you startup Tomcat:

# su - -s /bin/sh tomcat 

$ source /opt/tomcat-instance/sales.env 

$ $CATALINA_HOME/bin/shutdown.sh 

$ $CATALINA_HOME/bin/startup.sh

After Tomcat restarted, open the URL http://localhost/mysales (or use the server name instead of localhost) and you should see the "Execute Sales" link.
Clicking on this link should invoke the Java servlet and display "Executing Sales" in your browser.
If you are presented with an empty page instead, review the above steps and make sure you didn't miss a step.
Check also the log files under $CATALINA_BASE/logs.

Setting Up Second Tomcat JVM Instance

General

If you've gone through all the previous steps in this HOWTO, then the following steps should be very easy to follow and to understand
without much explanations. Therefore, I'll provide here just the steps for setting up a second Tomcat JVM instance and
an application called "Order".

Steps for Second Tomcat JVM Instance and Application

Login as root and execute the following steps to setup the second Tomcat JVM instance:

# mkdir -p /opt/tomcat-instance/order.example.com 

# cd /opt/tomcat-instance/order.example.com 

#

# cp -a /var/lib/apache-tomcat-6.0.18/conf . 

# mkdir common logs temp server shared webapps work 

#

# chown -R tomcat.tomcat /opt/tomcat-instance/order.example.com 

#

# su - -s /bin/sh tomcat 

$ cat > /opt/tomcat-instance/order.env << EOF

export JAVA_HOME=/usr/java/jdk1.6.0_10

export PATH=\$JAVA_HOME/bin:\$PATH

export CATALINA_HOME=/var/lib/apache-tomcat-6.0.18

export CATALINA_BASE=/opt/tomcat-instance/order.example.com

EOF 

$

$ source /opt/tomcat-instance/order.env 

$

For the second Tomcat JVM instance the default port numbers need to be changed in $CATALINA_BASE/conf/server.xml
(/opt/tomcat-instance/order.example.com/conf/server.xml).
In the following example I increased the port numbers by one:

    <Server port="8006" shutdown="SHUTDOWN">

 

    <Connector port="8081" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8444" />

 

    <Connector port="8010" protocol="AJP/1.3" redirectPort="8444" />

Create a new application root directory:

$ mkdir $CATALINA_BASE/webapps/order

To configure the new web application, edit $CATALINA_BASE/conf/server.xml
(/opt/tomcat-instance/order.example.com/conf/server.xml) and add the following entry in blue:

 

      <Host name="localhost"  appBase="webapps"

            unpackWARs="true" autoDeploy="true"

            xmlValidation="false" xmlNamespaceAware="false">

 

        <Context docBase="order" path="/myorder"/>

Create a new home page for the new "Order" application and include a link to the Java servlet that will be setup next:

$ cat > $CATALINA_BASE/webapps/order/index.html << EOF

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

"http://www.w3.org/TR/html4/loose.dtd">

<HTML>

<HEAD><META http-equiv=Content-Type content="text/html"></HEAD>

<BODY>

<H3>Apache Tomcat Order Home Page</H3>

<a href="/myorder/execute">Execute Order</a> 

</BODY>

</HTML>

EOF 

$

Now setup and create a new Java servlet:

$ mkdir -p $CATALINA_BASE/webapps/order/WEB-INF/classes 

$ mkdir $CATALINA_BASE/webapps/order/WEB-INF/lib

$ cat $CATALINA_BASE/webapps/order/WEB-INF/classes/Order.java 

import java.io.*;

import javax.servlet.*;

import javax.servlet.http.*;

 

public class Order extends HttpServlet {

 

    public void doGet(HttpServletRequest request, HttpServletResponse response)

    throws IOException, ServletException

    {

        response.setContentType("text/html");

        PrintWriter out = response.getWriter();

        out.println("<html>");

        out.println("<head>");

        out.println("<title>Order Page</title>");

        out.println("</head>");

        out.println("<body>");

        out.println("<h1>Executing Order ...</h1>");

        out.println("</body>");

        out.println("</html>");

    }

}

Compile the new Java servlet:

$ cd $CATALINA_BASE/webapps/order/WEB-INF/classes 

$ javac -classpath "$CATALINA_HOME/lib/*" Order.java 

$ ls 

Order.class  Order.java

$

Configure the Java servlet:

$ cat $CATALINA_BASE/webapps/order/WEB-INF/web.xml 

<?xml version="1.0" encoding="ISO-8859-1"?>

 

<web-app xmlns="http://java.sun.com/xml/ns/javaee"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"

    version="2.5">

 

  <servlet>

    <servlet-name>servlet_order</servlet-name>

    <servlet-class>Order</servlet-class>

  </servlet>

 

  <servlet-mapping>

    <servlet-name>servlet_order</servlet-name>

    <url-pattern>/execute</url-pattern>

  </servlet-mapping>

 

</web-app>

Now make sure to relogin as tomcat and start the second Tomcat JVM instance:

# su - -s /bin/sh tomcat 

$ source /opt/tomcat-instance/order.env 

$ $CATALINA_HOME/bin/startup.sh

After the second Tomcat JVM restarted, open the URL http://localhost:8081/myorder
(or use the server name instead of localhost) and you should see the "Execute Order" link.
Clicking on this link should invoke the Java servlet and display "Executing Order" in your browser.
If you are presented with an empty page instead, review the above steps and make sure you didn't miss a step.
Check also the log files under $CATALINA_BASE/logs.

Bibliography and References

Apache Tomcat

Tomcat: The Definitive Guide, 2nd Edition

Simple Tomcat Startup Script

2009-07-03T15:02:00.000+05:30

Simple Tomcat Startup Script:

#!/bin/sh
# Tomcat Startup Script

CATALINA_HOME=/mnt/apache-tomcat-6.0.20; export CATALINA_HOME
JAVA_HOME=/usr/java/jdk1.6.0_06; export JAVA_HOME
TOMCAT_OWNER=tomcat; export TOMCAT_OWNER

start() {
echo -n "Starting Tomcat: "
su -p -s /bin/sh tomcat $CATALINA_HOME/bin/startup.sh
sleep 2
}
stop() {
echo -n "Stopping Tomcat: "
su -p -s /bin/sh tomcat $CATALINA_HOME/bin/shutdown.sh
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo $"Usage: tomcat {start|stop|restart}"
exit
esac

Paes-Dlouhy in French Open final

2009-06-04T23:52:00.000+05:30

Leander Paes and Lucas Dlouhy entered the final of the men's doubles at the French Open after a thrilling victory over top seeds Daniel Nestor and Nenad Zimonjic in the semi-finals on Thursday.

The Indo-Czech pair needed two tie-breakers to quell the challenge of the Canadian-Serbian duo and emerge 7-6(4), 7-6(5) winners, and set up a meeting with South Africa's Wesley Moodie and Belgium's Dick Norman in the final.

This will be Paes's third French Open doubles final. On the earlier occasions, in 1999 and 2001, he went on to win the title with estranged partner Mahesh Bhupathi.

Unseeded Moodie-Norman staged a grand recovery after losing the first set 0-6 to shock the United States' Bryan twins Bob and Mike, seeded second, 0-6, 7-6 (5), 6-4 in the first semi-final.

Mostly used Practises to Speed up your website

2009-05-14T00:32:00.000+05:30

Minimize HTTP Requests

tag: content

80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requests required to render the page. This is the key to faster pages.

One way to reduce the number of components in the page is to simplify the page's design. But is there a way to build pages with richer content while also achieving fast response times? Here are some techniques for reducing the number of HTTP requests, while still supporting rich page designs.

Combined files are a way to reduce the number of HTTP requests by combining all scripts into a single script, and similarly combining all CSS into a single stylesheet. Combining files is more challenging when the scripts and stylesheets vary from page to page, but making this part of your release process improves response times.

CSS Sprites are the preferred method for reducing the number of image requests. Combine your background images into a single image and use the CSS background-image and background-position properties to display the desired image segment.

Image maps combine multiple images into a single image. The overall size is about the same, but reducing the number of HTTP requests speeds up the page. Image maps only work if the images are contiguous in the page, such as a navigation bar. Defining the coordinates of image maps can be tedious and error prone. Using image maps for navigation is not accessible too, so it's not recommended.

Inline images use the data: URL scheme to embed the image data in the actual page. This can increase the size of your HTML document. Combining inline images into your (cached) stylesheets is a way to reduce HTTP requests and avoid increasing the size of your pages. Inline images are not yet supported across all major browsers.

Reducing the number of HTTP requests in your page is the place to start. This is the most important guideline for improving performance for first time visitors. As described in Tenni Theurer's blog post Browser Cache Usage - Exposed!, 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these first time visitors is key to a better user experience.

top | discuss this rule

Use a Content Delivery Network

tag: server

The user's proximity to your web server has an impact on response times. Deploying your content across multiple, geographically dispersed servers will make your pages load faster from the user's perspective. But where should you start?

As a first step to implementing geographically dispersed content, don't attempt to redesign your web application to work in a distributed architecture. Depending on the application, changing the architecture could include daunting tasks such as synchronizing session state and replicating database transactions across server locations. Attempts to reduce the distance between users and your content could be delayed by, or never pass, this application architecture step.

Remember that 80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc. This is the Performance Golden Rule. Rather than starting with the difficult task of redesigning your application architecture, it's better to first disperse your static content. This not only achieves a bigger reduction in response times, but it's easier thanks to content delivery networks.

A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.

Some large Internet companies own their own CDN, but it's cost-effective to use a CDN service provider, such as Akamai Technologies, Mirror Image Internet, or Limelight Networks. For start-up companies and private web sites, the cost of a CDN service can be prohibitive, but as your target audience grows larger and becomes more global, a CDN is necessary to achieve fast response times. At Yahoo!, properties that moved static content off their application web servers to a CDN improved end-user response times by 20% or more. Switching to a CDN is a relatively easy code change that will dramatically improve the speed of your web site.

top | discuss this rule

Add an Expires or a Cache-Control Header

tag: server

There are two things in this rule:

For static components: implement "Never expire" policy by setting far future Expires header

For dynamic components: use an appropriate Cache-Control header to help the browser with conditional requests

Web page designs are getting richer and richer, which means more scripts, stylesheets, images, and Flash in the page. A first-time visitor to your page may have to make several HTTP requests, but by using the Expires header you make those components cacheable. This avoids unnecessary HTTP requests on subsequent page views. Expires headers are most often used with images, but they should be used on all components including scripts, stylesheets, and Flash components.

Browsers (and proxies) use a cache to reduce the number and size of HTTP requests, making web pages load faster. A web server uses the Expires header in the HTTP response to tell the client how long a component can be cached. This is a far future Expires header, telling the browser that this response won't be stale until April 15, 2010.

      Expires: Thu, 15 Apr 2010 20:00:00 GMT

If your server is Apache, use the ExpiresDefault directive to set an expiration date relative to the current date. This example of the ExpiresDefault directive sets the Expires date 10 years out from the time of the request.

      ExpiresDefault "access plus 10 years"

Keep in mind, if you use a far future Expires header you have to change the component's filename whenever the component changes. At Yahoo! we often make this step part of the build process: a version number is embedded in the component's filename, for example, yahoo_2.0.6.js.

Using a far future Expires header affects page views only after a user has already visited your site. It has no effect on the number of HTTP requests when a user visits your site for the first time and the browser's cache is empty. Therefore the impact of this performance improvement depends on how often users hit your pages with a primed cache. (A "primed cache" already contains all of the components in the page.) We measured this at Yahoo! and found the number of page views with a primed cache is 75-85%. By using a far future Expires header, you increase the number of components that are cached by the browser and re-used on subsequent page views without sending a single byte over the user's Internet connection.

top | discuss this rule

Gzip Components

tag: server

The time it takes to transfer an HTTP request and response across the network can be significantly reduced by decisions made by front-end engineers. It's true that the end-user's bandwidth speed, Internet service provider, proximity to peering exchange points, etc. are beyond the control of the development team. But there are other variables that affect response times. Compression reduces response times by reducing the size of the HTTP response.

Starting with HTTP/1.1, web clients indicate support for compression with the Accept-Encoding header in the HTTP request.

      Accept-Encoding: gzip, deflate

If the web server sees this header in the request, it may compress the response using one of the methods listed by the client. The web server notifies the web client of this via the Content-Encoding header in the response.

      Content-Encoding: gzip

Gzip is the most popular and effective compression method at this time. It was developed by the GNU project and standardized by RFC 1952. The only other compression format you're likely to see is deflate, but it's less effective and less popular.

Gzipping generally reduces the response size by about 70%. Approximately 90% of today's Internet traffic travels through browsers that claim to support gzip. If you use Apache, the module configuring gzip depends on your version: Apache 1.3 uses mod_gzip while Apache 2.x uses mod_deflate.

There are known issues with browsers and proxies that may cause a mismatch in what the browser expects and what it receives with regard to compressed content. Fortunately, these edge cases are dwindling as the use of older browsers drops off. The Apache modules help out by adding appropriate Vary response headers automatically.

Servers choose what to gzip based on file type, but are typically too limited in what they decide to compress. Most web sites gzip their HTML documents. It's also worthwhile to gzip your scripts and stylesheets, but many web sites miss this opportunity. In fact, it's worthwhile to compress any text response including XML and JSON. Image and PDF files should not be gzipped because they are already compressed. Trying to gzip them not only wastes CPU but can potentially increase file sizes.

Gzipping as many file types as possible is an easy way to reduce page weight and accelerate the user experience.

top | discuss this rule

Put Stylesheets at the Top

tag: css

While researching performance at Yahoo!, we discovered that moving stylesheets to the document HEAD makes pages appear to be loading faster. This is because putting stylesheets in the HEAD allows the page to render progressively.

Front-end engineers that care about performance want a page to load progressively; that is, we want the browser to display whatever content it has as soon as possible. This is especially important for pages with a lot of content and for users on slower Internet connections. The importance of giving users visual feedback, such as progress indicators, has been well researched and documented. In our case the HTML page is the progress indicator! When the browser loads the page progressively the header, the navigation bar, the logo at the top, etc. all serve as visual feedback for the user who is waiting for the page. This improves the overall user experience.

The problem with putting stylesheets near the bottom of the document is that it prohibits progressive rendering in many browsers, including Internet Explorer. These browsers block rendering to avoid having to redraw elements of the page if their styles change. The user is stuck viewing a blank white page.

The HTML specification clearly states that stylesheets are to be included in the HEAD of the page: "Unlike A, [LINK] may only appear in the HEAD section of a document, although it may appear any number of times." Neither of the alternatives, the blank white screen or flash of unstyled content, are worth the risk. The optimal solution is to follow the HTML specification and load your stylesheets in the document HEAD.

top | discuss this rule

Put Scripts at the Bottom

tag: javascript

The problem caused by scripts is that they block parallel downloads. The HTTP/1.1 specification suggests that browsers download no more than two components in parallel per hostname. If you serve your images from multiple hostnames, you can get more than two downloads to occur in parallel. While a script is downloading, however, the browser won't start any other downloads, even on different hostnames.

In some situations it's not easy to move scripts to the bottom. If, for example, the script uses document.write to insert part of the page's content, it can't be moved lower in the page. There might also be scoping issues. In many cases, there are ways to workaround these situations.

An alternative suggestion that often comes up is to use deferred scripts. The DEFER attribute indicates that the script does not contain document.write, and is a clue to browsers that they can continue rendering. Unfortunately, Firefox doesn't support the DEFER attribute. In Internet Explorer, the script may be deferred, but not as much as desired. If a script can be deferred, it can also be moved to the bottom of the page. That will make your web pages load faster.

top | discuss this rule

Avoid CSS Expressions

tag: css

CSS expressions are a powerful (and dangerous) way to set CSS properties dynamically. They're supported in Internet Explorer, starting with version 5. As an example, the background color could be set to alternate every hour using CSS expressions.

      background-color: expression( (new Date()).getHours()%2 ? "#B8D4FF" : "#F08A00" );

As shown here, the expression method accepts a JavaScript expression. The CSS property is set to the result of evaluating the JavaScript expression. The expression method is ignored by other browsers, so it is useful for setting properties in Internet Explorer needed to create a consistent experience across browsers.

The problem with expressions is that they are evaluated more frequently than most people expect. Not only are they evaluated when the page is rendered and resized, but also when the page is scrolled and even when the user moves the mouse over the page. Adding a counter to the CSS expression allows us to keep track of when and how often a CSS expression is evaluated. Moving the mouse around the page can easily generate more than 10,000 evaluations.

One way to reduce the number of times your CSS expression is evaluated is to use one-time expressions, where the first time the expression is evaluated it sets the style property to an explicit value, which replaces the CSS expression. If the style property must be set dynamically throughout the life of the page, using event handlers instead of CSS expressions is an alternative approach. If you must use CSS expressions, remember that they may be evaluated thousands of times and could affect the performance of your page.

top | discuss this rule

Make JavaScript and CSS External

tag: javascript, css

Many of these performance rules deal with how external components are managed. However, before these considerations arise you should ask a more basic question: Should JavaScript and CSS be contained in external files, or inlined in the page itself?

Using external files in the real world generally produces faster pages because the JavaScript and CSS files are cached by the browser. JavaScript and CSS that are inlined in HTML documents get downloaded every time the HTML document is requested. This reduces the number of HTTP requests that are needed, but increases the size of the HTML document. On the other hand, if the JavaScript and CSS are in external files cached by the browser, the size of the HTML document is reduced without increasing the number of HTTP requests.

The key factor, then, is the frequency with which external JavaScript and CSS components are cached relative to the number of HTML documents requested. This factor, although difficult to quantify, can be gauged using various metrics. If users on your site have multiple page views per session and many of your pages re-use the same scripts and stylesheets, there is a greater potential benefit from cached external files.

Many web sites fall in the middle of these metrics. For these sites, the best solution generally is to deploy the JavaScript and CSS as external files. The only exception where inlining is preferable is with home pages, such as Yahoo!'s front page and My Yahoo!.
Home pages that have few (perhaps only one) page view per session may find that inlining JavaScript and CSS results in faster end-user response times.

For front pages that are typically the first of many page views, there are techniques that leverage the reduction of HTTP requests that inlining provides, as well as the caching benefits achieved through using external files. One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser's cache.

top | discuss this rule

Reduce DNS Lookups

tag: content

The Domain Name System (DNS) maps hostnames to IP addresses, just as phonebooks map people's names to their phone numbers. When you type www.yahoo.com into your browser, a DNS resolver contacted by the browser returns that server's IP address. DNS has a cost. It typically takes 20-120 milliseconds for DNS to lookup the IP address for a given hostname. The browser can't download anything from this hostname until the DNS lookup is completed.

DNS lookups are cached for better performance. This caching can occur on a special caching server, maintained by the user's ISP or local area network, but there is also caching that occurs on the individual user's computer. The DNS information remains in the operating system's DNS cache (the "DNS Client service" on Microsoft Windows). Most browsers have their own caches, separate from the operating system's cache. As long as the browser keeps a DNS record in its own cache, it doesn't bother the operating system with a request for the record.

Internet Explorer caches DNS lookups for 30 minutes by default, as specified by the DnsCacheTimeout registry setting. Firefox caches DNS lookups for 1 minute, controlled by the network.dnsCacheExpiration configuration setting. (Fasterfox changes this to 1 hour.)

When the client's DNS cache is empty (for both the browser and the operating system), the number of DNS lookups is equal to the number of unique hostnames in the web page. This includes the hostnames used in the page's URL, images, script files, stylesheets, Flash objects, etc. Reducing the number of unique hostnames reduces the number of DNS lookups.

Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the page. Avoiding DNS lookups cuts response times, but reducing parallel downloads may increase response times. My guideline is to split these components across at least two but no more than four hostnames. This results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

top | discuss this rule

Minify JavaScript and CSS

tag: javascript, css

Minification is the practice of removing unnecessary characters from code to reduce its size thereby improving load times. When code is minified all comments are removed, as well as unneeded white space characters (space, newline, and tab). In the case of JavaScript, this improves response time performance because the size of the downloaded file is reduced. Two popular tools for minifying JavaScript code are JSMin and YUI Compressor. The YUI compressor can also minify CSS.

Obfuscation is an alternative optimization that can be applied to source code. It's more complex than minification and thus more likely to generate bugs as a result of the obfuscation step itself. In a survey of ten top U.S. web sites, minification achieved a 21% size reduction versus 25% for obfuscation. Although obfuscation has a higher size reduction, minifying JavaScript is less risky.

In addition to minifying external scripts and styles, inlined <script> and <style> blocks can and should also be minified. Even if you gzip your scripts and styles, minifying them will still reduce the size by 5% or more. As the use and size of JavaScript and CSS increases, so will the savings gained by minifying your code.

top | discuss this rule

Avoid Redirects

tag: content

Redirects are accomplished using the 301 and 302 status codes. Here's an example of the HTTP headers in a 301 response:

      HTTP/1.1 301 Moved Permanently
      Location: http://example.com/newuri
      Content-Type: text/html

The browser automatically takes the user to the URL specified in the Location field. All the information necessary for a redirect is in the headers. The body of the response is typically empty. Despite their names, neither a 301 nor a 302 response is cached in practice unless additional headers, such as Expires or Cache-Control, indicate it should be. The meta refresh tag and JavaScript are other ways to direct users to a different URL, but if you must do a redirect, the preferred technique is to use the standard 3xx HTTP status codes, primarily to ensure the back button works correctly.

The main thing to remember is that redirects slow down the user experience. Inserting a redirect between the user and the HTML document delays everything in the page since nothing in the page can be rendered and no components can start being downloaded until the HTML document has arrived.

One of the most wasteful redirects happens frequently and web developers are generally not aware of it. It occurs when a trailing slash (/) is missing from a URL that should otherwise have one. For example, going to http://astrology.yahoo.com/astrology results in a 301 response containing a redirect to http://astrology.yahoo.com/astrology/ (notice the added trailing slash). This is fixed in Apache by using Alias or mod_rewrite, or the DirectorySlash directive if you're using Apache handlers.

Connecting an old web site to a new one is another common use for redirects. Others include connecting different parts of a website and directing the user based on certain conditions (type of browser, type of user account, etc.). Using a redirect to connect two web sites is simple and requires little additional coding. Although using redirects in these situations reduces the complexity for developers, it degrades the user experience. Alternatives for this use of redirects include using Alias and mod_rewrite if the two code paths are hosted on the same server. If a domain name change is the cause of using redirects, an alternative is to create a CNAME (a DNS record that creates an alias pointing from one domain name to another) in combination with Alias or mod_rewrite.

top | discuss this rule

Remove Duplicate Scripts

tag: javascript

It hurts performance to include the same JavaScript file twice in one page. This isn't as unusual as you might think. A review of the ten top U.S. web sites shows that two of them contain a duplicated script. Two main factors increase the odds of a script being duplicated in a single web page: team size and number of scripts. When it does happen, duplicate scripts hurt performance by creating unnecessary HTTP requests and wasted JavaScript execution.

Unnecessary HTTP requests happen in Internet Explorer, but not in Firefox. In Internet Explorer, if an external script is included twice and is not cacheable, it generates two HTTP requests during page loading. Even if the script is cacheable, extra HTTP requests occur when the user reloads the page.

In addition to generating wasteful HTTP requests, time is wasted evaluating the script multiple times. This redundant JavaScript execution happens in both Firefox and Internet Explorer, regardless of whether the script is cacheable.

One way to avoid accidentally including the same script twice is to implement a script management module in your templating system. The typical way to include a script is to use the SCRIPT tag in your HTML page.

      <script type="text/javascript" src="menu_1.0.17.js"></script>

An alternative in PHP would be to create a function called insertScript.

      <?php insertScript("menu.js") ?>

In addition to preventing the same script from being inserted multiple times, this function could handle other issues with scripts, such as dependency checking and adding version numbers to script filenames to support far future Expires headers.

top | discuss this rule

Configure ETags

tag: server

Entity tags (ETags) are a mechanism that web servers and browsers use to determine whether the component in the browser's cache matches the one on the origin server. (An "entity" is another word a "component": images, scripts, stylesheets, etc.) ETags were added to provide a mechanism for validating entities that is more flexible than the last-modified date. An ETag is a string that uniquely identifies a specific version of a component. The only format constraints are that the string be quoted. The origin server specifies the component's ETag using the ETag response header.

      HTTP/1.1 200 OK
      Last-Modified: Tue, 12 Dec 2006 03:03:59 GMT
      ETag: "10c24bc-4ab-457e1c1f"
      Content-Length: 12195

Later, if the browser has to validate a component, it uses the If-None-Match header to pass the ETag back to the origin server. If the ETags match, a 304 status code is returned reducing the response by 12195 bytes for this example.

      GET /i/yahoo.gif HTTP/1.1
      Host: us.yimg.com
      If-Modified-Since: Tue, 12 Dec 2006 03:03:59 GMT
      If-None-Match: "10c24bc-4ab-457e1c1f"
      HTTP/1.1 304 Not Modified

The problem with ETags is that they typically are constructed using attributes that make them unique to a specific server hosting a site. ETags won't match when a browser gets the original component from one server and later tries to validate that component on a different server, a situation that is all too common on Web sites that use a cluster of servers to handle requests. By default, both Apache and IIS embed data in the ETag that dramatically reduces the odds of the validity test succeeding on web sites with multiple servers.

The ETag format for Apache 1.3 and 2.x is inode-size-timestamp. Although a given file may reside in the same directory across multiple servers, and have the same file size, permissions, timestamp, etc., its inode is different from one server to the next.

IIS 5.0 and 6.0 have a similar issue with ETags. The format for ETags on IIS is Filetimestamp:ChangeNumber. A ChangeNumber is a counter used to track configuration changes to IIS. It's unlikely that the ChangeNumber is the same across all IIS servers behind a web site.

The end result is ETags generated by Apache and IIS for the exact same component won't match from one server to another. If the ETags don't match, the user doesn't receive the small, fast 304 response that ETags were designed for; instead, they'll get a normal 200 response along with all the data for the component. If you host your web site on just one server, this isn't a problem. But if you have multiple servers hosting your web site, and you're using Apache or IIS with the default ETag configuration, your users are getting slower pages, your servers have a higher load, you're consuming greater bandwidth, and proxies aren't caching your content efficiently. Even if your components have a far future Expires header, a conditional GET request is still made whenever the user hits Reload or Refresh.

If you're not taking advantage of the flexible validation model that ETags provide, it's better to just remove the ETag altogether. The Last-Modified header validates based on the component's timestamp. And removing the ETag reduces the size of the HTTP headers in both the response and subsequent requests. This Microsoft Support article describes how to remove ETags. In Apache, this is done by simply adding the following line to your Apache configuration file:

      FileETag none

top | discuss this rule

Make Ajax Cacheable

tag: content

One of the cited benefits of Ajax is that it provides instantaneous feedback to the user because it requests information asynchronously from the backend web server. However, using Ajax is no guarantee that the user won't be twiddling his thumbs waiting for those asynchronous JavaScript and XML responses to return. In many applications, whether or not the user is kept waiting depends on how Ajax is used. For example, in a web-based email client the user will be kept waiting for the results of an Ajax request to find all the email messages that match their search criteria. It's important to remember that "asynchronous" does not imply "instantaneous".

To improve performance, it's important to optimize these Ajax responses. The most important way to improve the performance of Ajax is to make the responses cacheable, as discussed in Add an Expires or a Cache-Control Header. Some of the other rules also apply to Ajax:

Gzip Components
Reduce DNS Lookups
Minify JavaScript
Avoid Redirects
Configure ETags

Let's look at an example. A Web 2.0 email client might use Ajax to download the user's address book for autocompletion. If the user hasn't modified her address book since the last time she used the email web app, the previous address book response could be read from cache if that Ajax response was made cacheable with a future Expires or Cache-Control header. The browser must be informed when to use a previously cached address book response versus requesting a new one. This could be done by adding a timestamp to the address book Ajax URL indicating the last time the user modified her address book, for example, &t=1190241612. If the address book hasn't been modified since the last download, the timestamp will be the same and the address book will be read from the browser's cache eliminating an extra HTTP roundtrip. If the user has modified her address book, the timestamp ensures the new URL doesn't match the cached response, and the browser will request the updated address book entries.

Even though your Ajax responses are created dynamically, and might only be applicable to a single user, they can still be cached. Doing so will make your Web 2.0 apps faster.

top | discuss this rule

Flush the Buffer Early

tag: server

When users request a page, it can take anywhere from 200 to 500ms for the backend server to stitch together the HTML page.
During this time, the browser is idle as it waits for the data to arrive.
In PHP you have the function flush().
It allows you to send your partially ready HTML response to the browser so that
the browser can start fetching components while your backend is busy with the rest of the HTML page.
The benefit is mainly seen on busy backends or light frontends.

A good place to consider flushing is right after the HEAD because the HTML for the head is
usually easier to produce and it allows you to include any CSS and JavaScript
files for the browser to start fetching in parallel while the backend is still processing.

Example:

 
      ... <!-- css, js -->
    </head>
    <?php flush(); ?> 
    <body>
      ... <!-- content -->

Yahoo! search pioneered research and real user testing to prove the benefits of using this technique.

top

Use GET for AJAX Requests

tag: server

The Yahoo! Mail team found that when using XMLHttpRequest, POST is implemented in the browsers as a two-step process:
sending the headers first, then sending data. So it's best to use GET, which only takes one TCP packet to send (unless you have a lot of cookies).
The maximum URL length in IE is 2K, so if you send more than 2K data you might not be able to use GET.

An interesting side affect is that POST without actually posting any data behaves like GET.
Based on the HTTP specs, GET is meant for retrieving information, so it
makes sense (semantically) to use GET when you're only requesting data, as opposed to sending data to be stored server-side.

top

Post-load Components

tag: content

You can take a closer look at your page and ask yourself: "What's absolutely required in order to render the page initially?".
The rest of the content and components can wait.

JavaScript is an ideal candidate for splitting before and after the onload event. For example
if you have JavaScript code and libraries that do drag and drop and animations, those can wait,
because dragging elements on the page comes after the initial rendering.
Other places to look for candidates for post-loading include hidden content (content that appears after a user action) and images below the fold.

Tools to help you out in your effort: YUI Image Loader allows you to delay images
below the fold and the YUI Get utility is an easy way to include JS and CSS on the fly.
For an example in the wild take a look at Yahoo! Home Page with Firebug's Net Panel turned on.

It's good when the performance goals are inline with other
web development best practices. In this case, the idea of progressive enhancement tells us that JavaScript, when supported, can
improve the user experience but you have to make sure the page works even without JavaScript. So after you've made sure the page
works fine, you can enhance it with some post-loaded scripts that give you more bells and whistles such as drag and drop and animations.

top

Preload Components

tag: content

Preload may look like the opposite of post-load, but it actually has a different goal.
By preloading components you can take advantage of the time the browser is idle and request components
(like images, styles and scripts) you'll need in the future.
This way when the user visits the next page, you could have most of the components already in
the cache and your page will load much faster for the user.

There are actually several types of preloading:

Unconditional preload - as soon as onload fires, you go ahead and fetch some extra components.
Check google.com for an example of how a sprite image is requested onload. This sprite image is
not needed on the google.com homepage, but it is needed on the consecutive search result page.

Conditional preload - based on a user action you make an educated guess where the user is headed next and preload accordingly.
On search.yahoo.com you can see how some extra components are requested
after you start typing in the input box.

Anticipated preload - preload in advance before launching a redesign. It often happens after a redesign that you hear:
"The new site is cool, but it's slower than before". Part of the problem could be that the users were visiting your old site with a
full cache, but the new one is always an empty cache experience. You can mitigate this side effect by preloading some
components before you even launched the redesign. Your old site can use the time the browser is idle and request images and scripts
that will be used by the new site

top

Reduce the Number of DOM Elements

tag: content

A complex page means more bytes to download and it also means slower DOM access in JavaScript. It makes a difference
if you loop through 500 or 5000 DOM elements on the page when you want to add an event handler for example.

A high number of DOM elements can be a symptom that there's something that should be improved with the markup
of the page without necessarily removing content.
Are you using nested tables for layout purposes?
Are you throwing in more <div>s only to fix layout issues?
Maybe there's a better and more semantically correct way to do your markup.

A great help with layouts are the YUI CSS utilities:
grids.css can help you with the overall layout, fonts.css and reset.css
can help you strip away the browser's defaults formatting.
This is a chance to start fresh and think about your markup,
for example use <div>s only when it makes sense semantically, and not because it renders a new line.

The number of DOM elements is easy to test, just type in Firebug's console:

document.getElementsByTagName('*').length

And how many DOM elements are too many? Check other similar pages that have good markup.
For example the Yahoo! Home Page is a pretty busy page and still under 700 elements (HTML tags).

top

Split Components Across Domains

tag: content

Splitting components allows you to maximize parallel downloads. Make sure you're using
not more than 2-4 domains because of the DNS lookup penalty.
For example, you can host your HTML and dynamic content
on www.example.org
and split static components between static1.example.org and static2.example.org

For more information check
"Maximizing Parallel Downloads in the Carpool Lane" by Tenni Theurer and Patty Chi.

top

Minimize the Number of iframes

tag: content

Iframes allow an HTML document to be inserted in the parent document.
It's important to understand how iframes work so they can be used effectively.

<iframe> pros:

Helps with slow third-party content like badges and ads

Security sandbox

Download scripts in parallel

<iframe> cons:

Costly even if blank

Blocks page onload

Non-semantic

top

No 404s

tag: content

HTTP requests are expensive so making an HTTP request and getting a useless response (i.e. 404 Not Found)
is totally unnecessary and will slow down the user experience without any benefit.

Some sites have helpful 404s "Did you mean X?", which is great for the user
experience but also wastes server resources (like database, etc).
Particularly bad is when the link to an external JavaScript is wrong and the result is a 404.
First, this download will block parallel downloads. Next the browser may try to parse
the 404 response body as if it were JavaScript code, trying to find something usable in it.

top

Reduce Cookie Size

tag: cookie

HTTP cookies are used for a variety of reasons such as authentication and personalization.
Information about cookies is exchanged in the HTTP headers between web servers and browsers.
It's important to keep the size of cookies as low as possible to minimize the impact on the user's response time.

For more information check
"When the Cookie Crumbles" by Tenni Theurer and Patty Chi.
The take-home of this research:

Eliminate unnecessary cookies

Keep cookie sizes as low as possible to minimize the impact on the user response time

Be mindful of setting cookies at the appropriate domain level so other sub-domains are not affected

Set an Expires date appropriately. An earlier Expires date or none removes the cookie sooner, improving the user response time

top

Use Cookie-free Domains for Components

tag: cookie

When the browser makes a request for a static image and sends cookies together with the request,
the server doesn't have any use for those cookies. So they only create network traffic for no good
reason. You should make sure static components are requested with cookie-free requests. Create
a subdomain and host all your static components there.

If your domain is www.example.org, you can host your static components
on static.example.org. However, if you've already set cookies on the top-level domain
example.org as opposed to www.example.org, then all the requests to
static.example.org will include those cookies. In this case, you can buy a whole new domain, host your static
components there, and keep this domain cookie-free. Yahoo! uses yimg.com, YouTube uses ytimg.com,
Amazon uses images-amazon.com and so on.

Another benefit of hosting static components on a cookie-free domain is that some proxies might refuse to cache
the components that are requested with cookies.
On a related note, if you wonder if you should use example.org or www.example.org for your home page, consider the cookie impact.
Omitting www leaves you no choice but to write cookies to *.example.org, so for performance reasons it's best to use the
www subdomain and
write the cookies to that subdomain.

top

Minimize DOM Access

tag: javascript

Accessing DOM elements with JavaScript is slow so in order to have a more responsive page, you should:

Cache references to accessed elements

Update nodes "offline" and then add them to the tree

Avoid fixing layout with JavaScript

For more information check the YUI theatre's
"High Performance Ajax Applications"
by Julien Lecomte.

top

Develop Smart Event Handlers

tag: javascript

Sometimes pages feel less responsive because of too many event handlers attached to different
elements of the DOM tree which are then executed too often. That's why using event delegation is a good approach.
If you have 10 buttons inside a div, attach only one event handler to the div wrapper, instead of
one handler for each button. Events bubble up so you'll be able to catch the event and figure out which button it originated from.

You also don't need to wait for the onload event in order to start doing something with the DOM tree.
Often all you need is the element you want to access to be available in the tree. You don't have to wait for all images to be downloaded.

DOMContentLoaded is the event you might consider using instead of onload, but until it's available in all browsers, you
can use the YUI Event utility, which has an onAvailable method.

For more information check the YUI theatre's
"High Performance Ajax Applications"
by Julien Lecomte.

top

Choose <link> over @import

tag: css

One of the previous best practices states that CSS should be at the top in order to allow for
progressive rendering.

In IE @import behaves the same as using <link> at the bottom of the page, so it's best not to use it.

top

Avoid Filters

tag: css

The IE-proprietary AlphaImageLoader filter aims to fix a problem with semi-transparent true color PNGs in IE versions < 7.
The problem with this filter is that it blocks rendering and freezes the browser while the image is being downloaded.
It also increases memory consumption and is applied per element, not per image, so the problem is multiplied.

The best approach is to avoid AlphaImageLoader completely and use gracefully degrading PNG8 instead, which are fine in IE.
If you absolutely need AlphaImageLoader, use the underscore hack _filter as to not penalize your IE7+ users.

top

Optimize Images

tag: images

After a designer is done with creating the images for your web page, there are still some things you can try before you
FTP those images to your web server.

You can check the GIFs and see if they are using a palette size corresponding
to the number of colors in the image. Using imagemagick it's easy to check using

identify -verbose image.gif

When you see an image useing 4 colors and a 256 color "slots" in the palette, there is room for improvement.

Try converting GIFs to PNGs and see if there is a saving. More often than not, there is.
Developers often hesitate to use PNGs due to the limited support in browsers, but this is now a thing of the past.
The only real problem is alpha-transparency in true color PNGs, but then again, GIFs are not true color and don't
support variable transparency either.
So anything a GIF can do, a palette PNG (PNG8) can do too (except for animations).
This simple imagemagick command results in totally safe-to-use
PNGs:

convert image.gif image.png

"All we are saying is: Give PiNG a Chance!"

Run pngcrush (or any other PNG optimizer tool) on all your PNGs. Example:

pngcrush image.png -rem alla -reduce -brute result.png

Run jpegtran on all your JPEGs. This tool does lossless JPEG operations such as rotation and can also be used to optimize
and remove comments and other useless information (such as EXIF information) from your images.

jpegtran -copy none -optimize -perfect src.jpg dest.jpg

top

Optimize CSS Sprites

tag: images

Arranging the images in the sprite horizontally as opposed to vertically usually results in a smaller file size.

Combining similar colors in a sprite helps you keep the color count low, ideally under 256 colors so to fit in a PNG8.

"Be mobile-friendly" and don't leave big gaps between the images in a sprite. This doesn't affect the file size as much
but requires less memory for the user agent to decompress the image into a pixel map.
100x100 image is 10 thousand pixels, where 1000x1000 is 1 million pixels

top

Don't Scale Images in HTML

tag: images

Don't use a bigger image than you need just because you can set the width and height in HTML.
If you need

<img width="100" height="100" src="mycat.jpg" alt="My Cat" />

then your image (mycat.jpg) should be 100x100px rather than a scaled down 500x500px image.

top

Make favicon.ico Small and Cacheable

tag: images

The favicon.ico is an image that stays in the root of your server.
It's a necessary evil because even if you don't care about it the
browser will still request it, so it's better not to respond with a 404 Not Found.
Also since it's on the same server, cookies are sent every time it's requested.
This image also interferes with the download sequence, for example in IE when you request
extra components in the onload, the favicon will be downloaded before these extra components.

So to mitigate the drawbacks of having a favicon.ico make sure:

It's small, preferably under 1K.

Set Expires header with what you feel comfortable (since you cannot rename it if you decide to change it).
You can probably safely set the Expires header a few months in the future.
You can check the last modified date of your current favicon.ico to make an informed decision.

Imagemagick can help you create small favicons

top

Keep Components under 25K

tag: mobile

This restriction is related to the fact that iPhone won't cache components bigger than 25K.
Note that this is the uncompressed size. This is where minification is important
because gzip alone may not be sufficient.

For more information check "Performance Research, Part 5: iPhone Cacheability - Making it Stick" by Wayne Shea and Tenni Theurer.

top

Pack Components into a Multipart Document

tag: mobile

Packing components into a multipart document is like an email with attachments,
it helps you fetch several components with one HTTP request (remember: HTTP requests are expensive).
When you use this technique, first check if the user agent supports it (iPhone does not).

How To Save Traffic With Apache's mod_deflate

2009-05-14T00:18:00.002+05:30

In this tutorial I will describe how to install and configure mod_deflate on an Apache2 web server. mod_deflate allows Apache2 to compress files and deliver them to clients (e.g. browsers) that can handle compressed content which most modern browsers do. With mod_deflate, you can compress HTML, text or XML files to approx. 20 - 30% of their original sizes, thus saving you server traffic and making your modem users happier.

Compressing files causes a slightly higher load on the server, but in my experience this is compensated by the fact that the clients' connection times to your server decrease a lot. For example, a modem user that needed seven seconds to download an uncompressed HTML file might now only need two seconds for the same, but compressed file.

By using mod_deflate you don't have to be afraid that you exclude users with older browsers that cannot handle compressed content. The browser negotiates with the server before any file is transferred, and if the browser does not have the capability to handle compressed content, the server delivers the files uncompressed.

mod_deflate has replaced Apache 1.3's mod_gzip in Apache2. If you want to serve compressed files with Apache 1.3, take a look at this tutorial: mod_gzip - serving compressed content by the Apache webserver

1 Enable mod_deflate

If you have Apache2 installed, mod_deflate should also already be installed on your system. Now we have to enable it. On Linux with apache 2.2 installed, we can do it by setting the line in httpd.conf:

SetOutputFilter DEFLATE

Then restart Apache2:

/etc/init.d/httpd restart

On other distributions you might have to edit Apache2's configuration manually to enable mod_deflate. You might have to add a line like this to the LoadModule section:

LoadModule deflate_module /usr/lib/apache2/modules/mod_deflate.so

Make sure you adjust the path to mod_deflate.so, and restart Apache2 afterwards.

2 Configure mod_deflate

The compression of files can be configured in one of two ways: either explicit exclusion of files by extension or explicit inclusion of files by MIME type. You can enable mod_deflate for your whole Apache2 server, or just for specific virtual sites. Depending on this, either open

your Apache2's global server configuration section now or just the vhost configuration section where you want to enable mod_deflate.

2.1 Explicit Inclusion Of Files By MIME Type

If you want to compress HTML, text, and XML files only, add this line to your configuration:

AddOutputFilterByType DEFLATE text/html text/plain text/xml

This is the configuration I'm using because I don't want to compress images or PDF files or already compressed files such as zip files.

2.2 Explicit Exclusion Of Files By Extension

If you want to compress all file types and exclude just a few, you would add something like this to your configuration (instead of the line from section 2.1):

SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ \
    no-gzip dont-vary
SetEnvIfNoCase Request_URI \
    \.(?:exe|t?gz|zip|bz2|sit|rar)$ \
    no-gzip dont-vary
SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary

This would compress all files except images (gif, jpg, and png), already compressed files (like zip and tar.gz) and PDF files which makes sense because you do not gain much by compressing these file types.

2.3 Further Configuration Directives

Regardless whether you use the configuration from section 2.1 or 2.2, you should add these lines to your configuration:

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

These lines are for some older browsers that do not support compression of files other than HTML documents.

The configuration is now finished, and you must now restart Apache2. On Debian, you do it like this:

/etc/init.d/apache2 restart

To learn about further configuration directives, take a look at Apache Module mod_deflate.

3 Testing

To test our compression, we add a few directives to our mod_deflate configuration that log the compression ratio of delivered files. Open your mod_deflate configuration and add the following lines:

DeflateFilterNote Input input_info
DeflateFilterNote Output output_info
DeflateFilterNote Ratio ratio_info
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/apache2/deflate_log deflate

Make sure you replace /var/log/apache2 with your Apache2's log directory. This could be /var/log/httpd, /var/log/httpd2, etc.

Then restart Apache2. On Debian, do it like this:

/etc/init.d/apache2 restart

Now whenever a file is requested this will be logged in /var/log/apache2/deflate_log (or to whatever file you changed it to). A typical log line looks like this:

"GET /info.php HTTP/1.1" 7621/45430 (16%)

You see that the file info.php was requested and delivered. Its original size was 45430 bytes, and it was compressed to 7621 bytes or 16% of its original size! This is a great result, and if your web site mostly consists out of HTML, text, and XML files, mod_deflate will save you a lot of traffic, and for users with a low-bandwidth connection your site will load much faster.

If you don't need the logging after your tests anymore, you can undo the changes from section 3 and restart Apache2.

Over Rs 50,000 crore spent on Lok Sabha poll campaign

2009-05-12T17:44:00.001+05:30

Though attempts were made to check the spiralling amounts of money spent during the campaign for Lok Sabha polls, neither the political parties nor the Election Commission have attained the desired results in the run-up to the 2009 Parliamentary elections.

According to rough estimates, the actual cost of running the campaign has crossed the staggering sum of Rs 50,000 crore.

According to insiders in the Congress and the BJP, both parties spent over Rs 20 crore in merely ferrying their leaders across India during the campaign. While the Congress had hired 21 helicopters and 18 executive jets, the BJP had hired 21 helicopters and 14 jets. These parties had to shell out Rs 80,000 per hour on each helicopter, plus the landing charges.

Prime Minister Manmohan Singh , Congress president Sonia Gandhi and Bharatiya Janata Party's prime ministerial candidate L K Advani were allowed to use Air Force planes to travel to their election rallies across India.

The political parties have reportedly spent Rs 3000 crore on advertisement campaigns. The Congress spent Rs one crore just to buy the rights of superhit song Jai Ho, but it failed to click with the masses.

Even smaller parties like the Samajwadi Party and the Rashtriya Janata Dal also spent a considerable amount of money on the campaign trail.

The Election Commission's budget for the 2009 elections is Rs 1,300 crore, which includes the conduct of polls plus transportation and movement of security forces. The state governments and other government agencies have earmarked Rs 700 crore for photo identity cards, electronic voting machines and setting up polling booths.

Though attempts were made to check the spiralling amounts of money spent during the campaign for Lok Sabha polls, neither the political parties nor the Election Commission have attained the desired results in the run-up to the 2009 Parliamentary elections.

According to rough estimates, the actual cost of running the campaign has crossed the staggering sum of Rs 50,000 crore.

According to insiders in the Congress and the BJP, both parties spent over Rs 20 crore in merely ferrying their leaders across India during the campaign. While the Congress had hired 21 helicopters and 18 executive jets, the BJP had hired 21 helicopters and 14 jets. These parties had to shell out Rs 80,000 per hour on each helicopter, plus the landing charges.

Prime Minister Manmohan Singh , Congress president Sonia Gandhi and Bharatiya Janata Party's prime ministerial candidate L K Advani were allowed to use Air Force planes to travel to their election rallies across India.

The political parties have reportedly spent Rs 3000 crore on advertisement campaigns. The Congress spent Rs one crore just to buy the rights of superhit song Jai Ho, but it failed to click with the masses.

Even smaller parties like the Samajwadi Party and the Rashtriya Janata Dal also spent a considerable amount of money on the campaign trail.

The Election Commission's budget for the 2009 elections is Rs 1,300 crore, which includes the conduct of polls plus transportation and movement of security forces. The state governments and other government agencies have earmarked Rs 700 crore for photo identity cards, electronic voting machines and setting up polling booths.

Mayawati moves SC against Varun verdict

2009-05-12T17:40:00.001+05:30

The Uttar Pradesh government today moved the Supreme Court challenging advisory board's decision of revoking charges against BJP leader Varun Gandhi under the stringent NSA for his alleged hate speeches.

The advisory board on May 8 held that it neither found "plausible and convincing" grounds for the National Security Act being invoked against Varun nor was it satisfied by the explanation given by the Pilibhit District Magistrate.

29-year-old Varun, who is BJP's Lok Sabha candidate from Pilibhit, is currently on parole following a Supreme Court order after remaining in jail for nearly three weeks.

He was released from Etah jail on April 16. His parole expires on May 14. Varun was let off by a three-member Advisory board headed by senior judge of the Lucknow bench of Allahabad High Court Justice Pradeep Kant which went into the maintainability of Varun's detention by the UP government under the NSA imposed on March 29.
The Uttar Pradesh government today moved the Supreme Court challenging advisory board's decision of revoking charges against BJP leader Varun Gandhi under the stringent NSA for his alleged hate speeches.

The advisory board on May 8 held that it neither found "plausible and convincing" grounds for the National Security Act being invoked against Varun nor was it satisfied by the explanation given by the Pilibhit District Magistrate.

29-year-old Varun, who is BJP's Lok Sabha candidate from Pilibhit, is currently on parole following a Supreme Court order after remaining in jail for nearly three weeks.

He was released from Etah jail on April 16. His parole expires on May 14. Varun was let off by a three-member Advisory board headed by senior judge of the Lucknow bench of Allahabad High Court Justice Pradeep Kant which went into the maintainability of Varun's detention by the UP government under the NSA imposed on March 29.

Microsoft may lay off more if warranted: Ballmer

2009-05-12T17:39:00.000+05:30

Microsoft, which has announced laying off 5,000 employees including 55 in India, said on Tuesday it may look at more layoffs if the economic downturn dramatically worsens again.

"Presuming the economy hopefully stays as bad as it is and doesn't get dramatically worse, we will finish our plan, but if it gets dramatically worse again, we will look at things again," Microsoft Corporation CEO Steve Ballmer, told reporters in Mumbai.

The Redmond-based company had announced in January it would axe 5,000 jobs globally amid the ongoing slowdown.

It announced on Monday slashing one per cent of its 5,500-strong Indian workforce, amounting to 55 layoffs, in a bid to realign its business in the country.

It added that it would continue to hire and create employment opportunities in line with the recovery and growth of the Indian economy.

"We had said that we would lay-off about 5,000 people. We are still filling other jobs. We are mostly through that process globally and there is still some work to do," Ballmer said.

"There are areas where we are continuing to add people. As I said, these are global additions, so it is a little hard to separate our work globally from our work in India," he added.

Ballmer said Microsoft is the second largest foreign IT employer in India and he doesn't see a change in that.

In the second round of job cuts effected on May 5, the software major said it would lay off 3,000 employees. In January, Microsoft had laid off 1,350-1,400 people, largely in the US.

The Bill Gates-led firm said it would make strategic investments, which are best suited to the current economic environment.

Ballmer said Microsoft is the second largest foreign IT employer in India and he doesn't see a change in that.

In the second round of job cuts effected on May 5, the software major said it would lay off 3,000 employees. In January, Microsoft had laid off 1,350-1,400 people, largely in the US.

The Bill Gates-led firm said it would make strategic investments, which are best suited to the current economic environment.

vSphere 4: Forerunner to a Data Center Revolution?

2009-05-08T17:13:00.001+05:30

"The cloud" is a term that serves as a catchall for a variety of technology offerings that have Internet hosting as their common bond. Yet these services come in an unlimited variety of shapes and sizes. As the cloud begins to take on a less-wispy form, its potential is becoming clear. VMware imagines that it might one day function as a robust, complex virtual data center, with its own OS at the center.

Cloud computing has been a central subject and strategy for IT vendors of every sort, but the actual meaning of "cloud" remains hazy.

For Web-based information aggregators like Google (Nasdaq: GOOG) and Yahoo (Nasdaq: YHOO) , the cloud offers a mechanism for delivering advertising-driven content and services.

For Software as a Service (SaaS) vendors, including Salesforce.com (NYSE: CRM) , cloud infrastructures offer a highly efficient platform for hosting business applications and processes.

For service providers, the cloud provides the means for supporting emerging and yet-to-be-defined business and consumer offerings.

If one assumes there to be some cloud commonality among IT vendors, one would be largely incorrect. Not surprisingly, many storage vendors see the cloud as a way of supporting storage/data-centric consumer and business services.

Multi-platform systems vendors tend to define at least part of their cloud value propositions according to the capabilities of proprietary server platforms.

However, there are some similarities in x86/64 server-based cloud-specific products, largely because those systems are capable of commonly leveraging technologies from virtualization vendors such as VMware (NYSE: VMW), Microsoft (Nasdaq: MSFT) and Citrix (Nasdaq: CTXS) .

This last point provides the context for VMware's vSphere 4, which is essentially designed to drive forward the company's cloud computing strategy.

Driving Force

VMware has a different cloud vision: Rather than seeing a mechanism for simply delivering new or emerging service offerings, VMware imagines the enterprise data center as a highly flexible, scalable and changeable environment in which virtualization plays the central role in aggregating, integrating, managing and provisioning enormous pools of processor, server, storage and networking assets.

The company's view of the cloud appears to have struck a chord among its server vendor partners, whose executives offered their support on video or in person at the recent vSphere launch event at VMware headquarters in Palo Alto, Calif.

Cisco's (Nasdaq: CSCO) John Chambers, Dell's (Nasdaq: DELL) Michael Dell, EMC's (NYSE: EMC) Joe Tucci, HP's (NYSE: HPQ) James Munton and Intel's (Nasdaq: INTC) Pat Gelsinger were palpably enthusiastic about vSphere, and with good reason. If the effort succeeds to the extent that VMware and others expect, it will provide the driving force behind next-generation data centers.

Why? Because vSphere 4 is not just about cloud computing. While x86/64-based solutions have led server volume sales for several years, their overall performance and utilization have tended to suffer in comparison to Unix and mainframe systems.

Virtualization has helped to correct the traditionally woeful system utilization of x86/64 servers. Indeed, without virtualization, x86/64-based technologies would not be sustainable data center solutions.

Ushering In a New Age

VMware produced some eye-opening vSphere performance metrics -- including sustained 300,000 IOPS and up to 9,000 transactions per second on single systems -- that suggest a fundamental shift in performance that will allow x86/64 systems to fully inhabit every corner of the enterprise.

This, combined with other new features, including VMware Fault Tolerance, makes the platform eminently suitable for supporting business-critical applications and what VMware CEO Paul Maritz calls the "Big Computer" and the "21st Century Mainframe."

In other words, VMware considers vSphere 4 the key to ushering in an age in which highly virtualized, highly integrated industry standard x86/64 systems take over the jobs currently held by legacy enterprise systems.

Is this scenario remotely possible? Perhaps so. One could point to the emergence of x86/64 as the platform of choice in the vast majority of supercomputing installations -- a market once dominated by proprietary systems and technologies -- as an example of what is possible with innovative x86/64 development.

Is vSphere 4, then, poised to initiate the coming data center revolution?

Not quite. It is highly powerful and flexible, but VMware's new offering is a work in progress -- even though it is definitely several steps ahead of previous company offerings.

That said, if VMware delivers as promised on its product road map, vSphere 4 could become the platform to beat in x86/64 virtualization, and it will play an elemental role in how the company's customers and partners design, develop and deploy 21st century cloud computing data centers.

Gadgets on the Run: Keeping Tabs on Moving IT Assets

2009-05-08T17:07:00.002+05:30

As enterprises deploy growing ranks of mobile, remote and telecommuting employees, keeping track of the many mobile devices they use has introduced new headaches for IT managers. Mobile asset management is a part of the larger IT asset management strategy and should not be viewed as a separate type of asset management program.

It's no secret that the business world is going mobile. Companies are beginning to move to a virtual environment -- especially sales teams who have to be on the road more often. The trend of telecommuting is also starting to catch on. With this in mind, IT managers are faced with a new headache: managing what they can't see. Companies are struggling to create and understand mobile IT strategy.

IT asset managers are faced with the challenge of managing mobile "moving assets" in a variety of scenarios including, but not limited to:

geographically dispersed offices (including multi-national organizations)

larger number of mobile devices than ever before

growing trends toward telecommuting

virtualization technologies and ASP (hosted) services

While mobile and remote workforces are nothing new, moving assets can cost companies millions (even billions) of dollars when not accounted for, and they can put organizations at even greater risk. While physical hardware usually has a shelf life of three to five years, mobile assets have a much shorter lifespan.

Special Attention

There are a number of activities that enterprise IT staffs need to pay special attention to if they want to effectively manage these moving assets:

Define your mobile IT strategy. The tendency is to start with the technology and work backwards to try to solve the business problem. Accounting for mobile assets within your organization's regular ITAM (IT asset management) program should be the norm. However, mobile assets are likely the most difficult to track and manage, and -- most often -- the type of asset that is most often unaccounted for.

Identify the right tool set for managing mobile IT. There are many solutions out there, from scanning tagged hardware to automating software asset management solutions. There are many options available, but finding the right one is imperative. Ideally, whatever type of solution you use, it would be best to integrate it with other systems, such as your financial systems, to provide better trend and cost analysis, geographic mapping and more robust reporting, as well as to enable better ROI (return on investment) tracking against monies spent. Some tools that are helpful for managing mobile assets:
- Automated Solutions. Like IT asset management, mobile asset management needs discovery and ongoing management. From tracking hardware -- such as PCs, Macs, laptops, PDAs, smart phones and many other moving assets -- to tracking the software and user license sitting within the hardware -- from MS Office Suite to Adobe (Nasdaq: ADBE) -- having an automated asset tracking system can eliminate costly and time-consuming site visits. In the case of moving assets, an automated ITAM tracking system with a discovery feature would be extremely useful for reporting enterprise software and hardware assets on each of the moving assets without doing a physical "roll call."
- RFID. For the large enterprise, this can be extremely useful in tracking hardware -- from PDAs (personal data assistants) to laptops to servers. When the information from the barcode/scanning technology is integrated into the whole ITAM program, it can initially give you some powerful ROI.

Deal with leased and inactive mobile devices. How often do you realize that one of your employees has three BlackBerrys or two laptops? (Of course, he or she is only using one, while the other is propping the office door open). A simple solution is to run a quarterly report for missing leased equipment -- from laptops to mobile PDAs -- and review which computers have not connected to the network within 30 days or more. Assuming that you're following ITAM best practices, you'll be able to quickly find your missing and inactive items for redeployment within your organization or to retire the IT asset (instead of letting it become a dust catcher).

Manage software licenses -- reduce cost and decrease your risk. Software licenses on mobile assets allow vendors to pull in millions of dollars during vendor audits. Software licenses are usually licensed by usage and tracked via discovery. Similar to mobile devices, unused licenses should also be taken into account via usage analysis. By proactively managing software licenses and usage on mobile assets, you can reduce risk and costs.

Get in step with the greening of technology. While green technology does exist, it's in its infancy and it's expensive. However, IT disposal -- especially of mobile devices -- can be utilized for a company's green program (if one exists) or even for charitable donations (there are a lot of programs taking cell phones, laptops, PDAs and the like). Fundamentally, disposing of your mobile assets could be looked upon as a way of "doing good."

Bigger Picture

If nothing else, remember these three things:

Effective management of mobile assets is a part of the larger IT asset management strategy and should not be viewed as a separate type of asset management program.

One major advantage of proper management of mobile devices -- as well as fixed assets in general -- is the ability to show a quick return on investment in terms of dollars, savings and business gaps throughout the organization, especially with multi-location and multinational companies. This always scores points with the senior executives and may put some dollars back in the budget, as you can build a real case on savings with real dollars. Mobile devices certainly would account for a percentage of savings from retaining, recovering, deploying or retiring "moving" IT assets.

Because mobile devices are not always top of mind with the IT folks or senior management and rarely (if ever) tracked properly, it's imperative to create policies and processes -- and enforce them.

Tata, SBI among world's most reputable firms

2009-05-08T16:50:00.001+05:30

The Tata Group has been named the world's eleventh most reputed company, according to a study compiled by United States-based Reputation Institute.

Not just that, the Tata Group, whose Global Pulse Score was put at 80.89, has been ranked above global giants like Google, Microsoft, General Electric, Toyota, Coca-Cola, Intel, Univler, et cetera.

The Reputation Institute's Global Pulse is a measure of corporate reputation calculated by averaging perceptions of four main indicators -- trust, esteem, admiration, and good feeling -- obtained from a representative sample of at least 100 respondents in the companies' home countries. The Global Pulse scores are on a scale of 0 to 100.

Tata Group is one of India's largest industrial conglomerates and runs more than 98 firms.

For the record, Italian confectioner Ferrero has been ranked the world's most reputable business entity.

Is Barack Obama right about Bangalore?

2009-05-07T17:13:00.001+05:30

Before anyone in India gets hot under the collar about US President Barack Obama's tax proposals, because they might seem targeted at job creation in 'Bangalore,' it is important to understand what he is trying to do. For, on any rational basis, it is hard to be critical.

American companies that invest abroad have been tax-exempt on the profits from such businesses until they bring the profits back into the US; however, they have been allowed to claim a set-off on the expenses related to such investment.

This has been an open invitation to invest overseas and not in the home market, especially if the money is routed through tax havens so that the firms pay no tax on their profits anywhere. Mr Obama has called this a 'scam,' a term to which American businessmen have taken umbrage, but it is hard to think of it in any other terms.

The figures trotted out, showing that effective tax rates on such investments have been in the 2-3 percentage points range, support the president's drive to raise the effective level of tax on such corporate activity, at a time when he is running a gigantic deficit and needs money for other programmes.

Dhoni to lead Team India in T20 World Cup

2009-05-04T18:51:00.000+05:30

Mahendra Singh Dhoni will lead a 15-member Indian squad in the ICC Twenty20 World Championship in England next month.

Virender Sehwag has been named the vice captain for the series.

Apart from Dhoni and Sehwag, Paceman R P Singh was on Monday rewarded for his consistent performance in the ongoing IPL with a recall while fellow speedster Munaf Patel was dropped from India's 15-member squad for next month's Twenty20 World Cup in England .

Wicketkeeper-batsman Dinesh Karthik , who figured in India's squad that played the last Twenty20 in New Zealand in February end, has also been dropped from the squad which has no major surprises. Hard-hitting batsman Robin Uthappa was also omitted.

India, who won the inaugural edition of the championship in South Africa in 2007, will be led by Mahendra Singh Dhoni while Virender Sehwag has been named his deputy.

The squad includes five specialist batsmen, five specialist pacers, two spinners, two all-rounders and a wicket-keeper in Dhoni.

The selectors had earlier announced a list of 30 probables for the event, to be held in England from June 5 to 21, and today pruned it to the final 15. All participating teams have to submit the final squad by Tuesday as per the ICC rules.

As expected, there are no major surprises in the squad which was picked up by the selection panel, headed by former India captain Krishnamachari Srikkanth via teleconference.

Among others who could not make it to the final 15 are Tamil Nadu opener M Vijay, Mumbai trio of Ajinkya Rahane, Dhawal Kulkarni and Abhishek Nayar , Tamil Nadu batsman S Badrinath, Delhi batsman Virat Kohli , Bengal duo of Manoj Tiwary and Wriddhiman Saha, Haryana leggie Amit Mishra , Tamil Nadu pacer L Balaji and Madhya Pradesh stumper Naman Ojha.

Tamil Nadu off-spinner R Ashwin, who was named back-up for Harbhajan Singh , also failed to survive the pruning exercise.

Youngsters Abhishek Raut, who plays for Rajasthan Royals , and Bangalore's Shrivats Goswami also could not make the cut despite their decent showing in the Indian Premier League .

Joginder Sharma, who is remembered for his last over in the inaugural edition of the Twenty20 World Cup, could not find a place in the 30-member list of probables, along with Piyush Chawla and S Sreesanth .

Sreesanth is recovering from a back injury and has not played competitive cricket for the last few months.

Sachin Tendulkar had opted out of the Twenty20 World Cup in 2007 and continues to remain away from the shortest format of the game despite his good form in recent time.

Squad: M S Dhoni (c), Virender Sehwag (v-c), Gautam Gambhir , Suresh Raina , Yuvraj Singh , Yusuf Pathan , Rohit Sharma , Harbhajan Singh, Zaheer Khan , Ishant Sharma , Praveen Kumar, RP Singh, Ravindra Jadeja , Pragyan Ojha and Irfan Pathan .

India orders 250,000 OLPC laptops

2009-05-03T13:50:00.002+05:30

The '$10, world's cheapest laptop', developed in India has been given a quiet burial with the government placing an order for 250,000 XO laptops from the Nicholas Negroponte-led One Laptop Per Child (OLPC) Foundation.

The $10 'laptop' had turned into a major bone of contention with the global IT industry and experts blasting the device that was earlier projected as a challenger to the $100 laptop of the OLPC project.

Meanwhile, Satish Jha, OLPC India president and CEO, was quoted in the media as saying that the OLPC XO laptops "have been ordered for 1,500 schools (throughout the country) and the deliveries will begin in June."

Now lets have a look on the fake scheme of government ignoring the below key points:

This is one of the biggest fraud waiting to come.

1. Whats point in procuring laptop for schoolkids when p[rimary education DO NOT need any computer / laptop.

2. If at all laptops are baught, which operaing system it will work on? Who will pay license fee?

3. WHat applications will be there on laptop for use ? WHat value will they add ?

4. If procured, how distribution will take place? Why only 1500 schools, what about other thousands of schools?

5. We dont give funds for primary education to rural areas. Why waste funds for unnecessary laptops in urban areas?

This is just to eat commission. FRAUD. FRAUD. FRAUD. FRAUD.

Harshad Mehta's Reborn Version: Nirmal Kotecha and his Techniques

2009-05-02T13:43:00.002+05:30

The man who follows Harshad Mehta's technique:Nirmal Kotecha

The son of an LIC agent who also ran a medical shop in Kochi, Nirmal Kotecha hasn't done too badly for himself. At 32, he is estimated to be worth about Rs 500 crore (Rs 5 billion).

His acquaintances have many adjectives for him: From 'genius' to a man wildly passionate about the stock markets to someone who was in a tearing hurry to make money etc.

Last week, he earned another sobriquet -- 'the mastermind and the main beneficiary of the Pyramid Saimira forgery'-- from the capital market regulator.

The April 23 order by the regulator said Kotecha had masterminded the forgery of a Securities and Exchange Board of India letter ordering directors of Pyramid, which runs a chain of theatres, to make an open offer to shareholders.

Since the forgery was aimed at manipulating the company's share price, Sebi barred 230 people and entities from trading.

The regulator also suspects that Kotecha used several front companies to trade in various stocks, and that there are indications of massive fund rotation among these front entities.

In a way, this wasn't a surprise. Kotecha has had the distinction of being on the Sebi's watch list on at least two other price manipulation cases -- Atlanta Ltd and SEL Manufacturing.

The modus operandus in both cases was similar to that of Pyramid: As Sebi discovered, Kotecha bought stakes in the companies at throwaway prices from the promoters, rigged the prices and then dumped the stock.

This may sound boringly familiar; many operators do just this frequently in the Indian stock markets, but Kotecha did it with finesse that would have made his self-confessed inspiration -- Harshad Mehta -- proud. He started investing in the market in 1993 at the age of 16, when Harshad Mehta's scam came to light and the young Kotecha was one of his ardent admirers.

By the time Mehta's cookie crumbled, Kotecha was deeply into investing and became a sub-broker at the Kochi Stock Exchange at the age of 18. Kotecha made his real big money during the technology boom in 2000.

He shifted to the Mumbai market soon after.

In the process, he opened many companies -- Skyz Financial Consultant and Kotecha Capital were just two of them.

As Sebi found, he was also using a large number of front accounts, including those of his relatives, to manipulate the securities market and to route the funds through several layers -- a reason the regulator has requested the Reserve Bank of India, Financial Intelligence Unit and the income tax department to look into possible money laundering.

All through, Kotecha seems to have used his early contacts with many promoters of small gems and jewellery companies well (he had invested in many of their IPOs as well). No wonder 43 of the 230 entities in the Sebi order belong to the gems and jewellery sector.

Though he has been known to be dealing in small IPOs to make mega bucks, many agree he perhaps went too far this time.

First, he forged a Sebi letter in which the regulator directed the promoters to make an open offer for Pyramid Saimira -- potential market-moving information.

Then he and his partners in the Pyramid case also planted a fake company secretary and gave this person's number to journalists who were sent the forged letter. When journalists called for confirmation, this person impersonating the company secretary claimed that Pyramid had indeed received such a letter from the regulator!

So when the Pyramid stock price surged after the forged letter became public, Kotecha went for the kill and reduced his holding from 24 per cent to just 0.24 per cent in just three months, making a massive profit.

Of late, Kotecha had shifted his attention to private equity too. For example, his PE firm Kotecha Capital picked up 49 per cent stake in the Bangalore-based US Pizza.

Although the exact valuation of the deal isn't known, media reports said Kotecha will invest over Rs 500 crore (Rs 5 billion), including debt, as the fast food chain plans to expand at a furious pace.

But many people say it would be a mistake to write off Kotecha. After all, Sebi's order is just an interim one and Kotecha will obviously challenge it. He was let off on two earlier occasions. Will it be third time lucky for the market player?

What is Swine FLU ?

2009-05-02T13:32:00.002+05:30

Swine influenza (also called swine flu, hog flu, and pig flu) refers to influenza caused by those strains of influenza virus that usually infect pigs and are called swine influenza virus (SIV). Swine influenza is common in pigs in the midwestern United States (and occasionally in other states), Mexico, Canada, South America, Europe (including the United Kingdom, Sweden, and Italy), Kenya, Mainland China, Taiwan, Japan and other parts of eastern Asia.

Transmission of SIV from pigs to humans is not common. When it results in human influenza, it is called zoonotic swine flu. People who work with pigs, especially people with intense exposures, are at risk of catching swine flu. However, only about fifty such transmissions have been recorded since the mid-20th Century, when identification of influenza subtypes became possible. (Importantly, eating pork does not pose a risk of infection.) Rarely, these strains of swine flu can pass from human to human. In humans, the symptoms of swine flu are similar to those of influenza and of influenza-like illness in general, namely chills, fever, sore throat, muscle pains, severe headache, coughing, weakness and general discomfort.

The 2009 flu outbreak in humans that is widely known as "swine flu" is due to a new strain of influenza A virus subtype H1N1 that was produced by reassortment from one strain of human influenza virus, one strain of avian influenza virus, and two separate strains of SIV. The origin of this new strain is unknown, and the World Organization for Animal Health (OIE) reports that this strain has not been isolated in pigs.[2] It passes with apparent ease from human to human, an ability attributed to an as-yet unidentified mutation.[3] This 2009 H1N1 strain causes the normal symptoms of influenza, such as fever, coughing and headache.

HP unviels PRO-BOOK laptop Line

2009-04-29T12:17:00.001+05:30

Hewlett-Packard Co , the world’s top PC maker, is launching a new line of inexpensive business laptops with fresh features targeting users at small and medium-size companies.
The HP ProBook s-series, which starts shipping globally on Tuesday, is the company’s new mainstream business notebook, following the release of its higher-end, lightweight EliteBook line last year.
The ProBook replaces the HP Compaq line, although the Compaq name will continue to be used as a master brand name in other PCs.
The ProBook offers users a number of new features, including an optional Linux-based operating system pre-installed -- Novell Inc’s SuSE Linux Enterprise Desktop 11 -- for those seeking an alternative to the dominant Microsoft Corp Windows platform.
It is HP’s first-ever Linux pre-install on a standard business laptop, the company said. The PC maker does offer some netbooks with .
“It’s pretty much a natural evolution,” said Carol Hess- Nickels, HP’s director of marketing for worldwide business notebooks.
“We want to provide a different option ... it’s probably a little time yet before we’ll know exactly what the demand is, but we did think it was something worth trying.”
The ProBooks come with 14-inch, 15.6-inch and 17.3-inch screen sizes, with prices starting at $529. In another first for an HP business notebook, buyers will be able to add a color finish -- “merlot” -- if they choose.
Some models will also feature Qualcomm Inc’s Gobi technology, allowing them to use a single module to access different mobile broadband network technologies and mobile operators.
HP will also bring higher-end durability features, like its 3D DriveGuard -- which protects the hard drive if a laptop is dropped -- and a spill-resistant keyboard to the ProBook.
HP is the world’s No. 1 PC vendor, with a first-quarter global market share of more than 20 percent, according to research house IDC, well ahead of second-place Dell Inc. HP also took over the top spot in the US market from Dell in the first quarter.
The shares of Palo Alto, California-based HP closed the regular session down 35 cents at $35.45 on the New York Stock Exchange on Monday.

The Negative Voting Option: Elections

2009-04-29T12:02:00.001+05:30

In the forthcoming general election, as has become the practice in India, the choice you will probably get to exercise will be between voting for either Tweedledum or Tweedledee. It’s bad enough that it’s barely a choice. Worse still, though, both Tweedledum and Tweedledee may turn out to be murderers or rapists, as has been the reality in some constituencies of late. Vote we must, but the question is for whom, if the choice happens to be between the devil and the deep sea.

Why do we have to choose between two evils? Behind the noisy and colourful facade of elections, political parties decide on which candidates they will field regardless of the background, criminal or otherwise, of the candidate. The most recent example of this phenomenon is from the Tamar constituency in Jharkhand which earned laurels for defeating Shibu Soren in January. What is perhaps not widely known is that Gopal Krishna Patar, who defeated Soren, is an accused out on bail; Patar faces a criminal case under various sections of the Indian Penal Code, including attempt to murder.
What could the voters in the Tamar constituency do if they did not want to vote for either Soren, found guilty of murder in 2006, or Patar? There is no option currently; but there could easily be one. This option is popularly called “None of the above”, or Nota.
Though the idea of Nota captured the popular imagination after the citizen protests in the wake of 26/11, it is not new. The Law Commission of India first recommended it in May 1999, in its 170th report on electoral reforms.
In its report, the Law Commission submitted its recommendation for Nota in combination with the electoral requirement that a candidate gain at least 50%+1 of the votes cast to be declared the winner. These provisions, according to the commission, would go “a long way in ensuring purity of elections, keeping out criminals and other undesirable elements and also serve to minimize the role and importance of caste and religion”.
The report noted that such provisions would achieve two objectives. The first would be “to cut down or, at any rate, to curtail the significance and role played by caste factor in the electoral process… This means that a candidate has to carry with him several castes and communities, to succeed.” This would certainly work to reduce the caste-based fragmentation of the polity and help develop holistic and pluralistic perspectives.

The second objective would be “to put moral pressure on political parties not to put forward candidates with undesirable record i.e., criminals, corrupt elements and persons with unsavoury background… It also acts as a powerful disincentive against voter intimidation.” Given that the last election put 125 candidates with pending criminal cases into the Lok Sabha, discouraging candidates with dubious backgrounds is essential.
There are, no doubt, practical difficulties in implementing these provisions, which the Law Commission observed. “If electronic voting machines (EVM) are introduced throughout the country, it will become a little more easier to implement this,” the report said.
The Election Commission of India supported these suggestions in its recommendations to the government on 10 December 2001, and reiterated then again in a letter from the then chief election commissioner to the Prime Minister on 4 July 2004.
Despite such clear and specific recommendations, and having had electronic voting machines in use for quite a few years, the government has not considered it fit to implement this provision. The Law Commission seems to have foreseen this when it said in 1999 that, “problems arise because of...lack of requisite standards of behaviour and also of cooperation and understanding among the political parties to ensure a peaceful poll. As a matter of fact, the election offences are not decreasing but are increasing, with every passing election.” Technology and a maturing democracy are supposed to make things easier; but it’s the reverse in India.
Voters having to vote without having a real choice is not really democratic.
It is under the above circumstances that an option of “None of the above” or “I do not vote for any of the above candidates” has the potential of giving voters some real choice, thus taking us closer to real democracy. It can nudge political parties to select better candidates. In case the Nota option gets the highest number of votes cast, the law would require repolling, that too with the earlier candidates not being allowed to recontest. There will be some costs to repolls, though much less now with EVMs. But democracy needs and deserves such investments. If the return is an improvement in the quality of candidates, the investment would be well worth it.
Even without a repoll, some moral pressure may be applied on political parties. When the Nota option is repeatedly exercised across India, parties are sure to learn the lesson. The purpose of the exercise is not to ask voters to “not vote”, but rather to nudge political parties to select better candidates.

Gigabyte Launches 3 WinMob Handsets in India

2009-04-27T13:05:00.002+05:30

In Feb of this year, GIGABYTE Communications Inc. had announced its foray into the rapidly growing mobile phone market with the launch of its award winning new generation GSmart phones that included the MS820 touchscreen handset with GPS. They have now introduced two new models of touch screen Windows Mobile PDA phones for the Indian market along with the MS820 and - the MW702 and MW700.

The GSmart mobile phones being launched in India run on a Windows Mobile 6.1 Professional OS and come equipped with features that also include built-in GPS, Autofocus cameras some of which include Face Detection, Wi-Fi and Bluetooth as well as external memory support via microSD cards.

The GSmart MS820, according to the company is capable of working virtually with all audio, video and image formats and features -

3.5G: HSDPA 7.2Mbps, WCDMA 2100, EDGE, GPRS, GSM Tri-band
Modem: QUALCOMM MSM 6280
CPU: Marvell PXA270, 520MHz
2.8-inch VGA TFT LCD touch panel
5M camera with Face Detection
Smart sensor accelerometer
Built-in GPS: SiRF Star III

The MW702 comes with –

2.75G: EDGE, GPRS, GSM 900, 1800, 1900
CPU: Marvell PXA270, 520MHz
Modem: Qualcomm 6235
2.8-inch full touch panel with high resolution screen
“Smart Touch” Personal finger-touch UI
3.0 Mega pixels with Auto Focus Camera
Built-in GPS: SiRF III Built-in

The third in the set, the MW700 will feature –

EDGE, GPRS, GSM tri-band
Built-in GPS: SiRF Star III
14.95mm thin
Video Telephony
Modem: QUALCOMM MSM6235
CPU: Marvell PXA270, 520MHz
2.8-inch TFT LCD touch panel
2M camera with auto focus

The Smart Touch interface will allow easy management of the main function menu while the multimedia center has a photo editor, music player and FM radio. The GPS tracking feature enables users to download online maps and edit traveling paths to create personal traveling journals. Photographs can be tagged with GPS to record the location where they were taken.

The MS820, MW702 and MW700 are also available in the country for purchase and are priced at Rs. 36, 999, Rs. 29, 999 and Rs. 16, 999 respectively.

Yahoo Pulls the Plug on GeoCities

2009-04-27T13:02:00.001+05:30

Yahoo Inc is shutting down GeoCities, a free service that hosts personal home pages for consumers, which it acquired for more than $4 billion 10 years ago during the heyday of the dotcom boom. A posting on a Yahoo Help page for GeoCities on Thursday said the service was no longer accepting new customers and that it will be closing later this year, with more details about how individuals can save their data coming this summer. The move comes a few days after Yahoo said it would lay off nearly 700 workers, or 5 percent of its workforce.

Since Chief Executive Carol Bartz took the reins in January, Yahoo has pruned various products and properties to cut costs and focus on fundamentals, as it seeks to revive growth in a tough economy and fierce competition from Google Inc. Last week, Yahoo said it was shutting down Jumpcut, an online service for editing videos. Yahoo acquired GeoCities in 1999 in a stock deal valued at roughly $4.6 billion, Reuters reported at the time.
GeoCities was among the first companies to build online communities, with more than 3.5 million websites hosted on its service in the late 1990s. But GeoCities fell out of favor in recent years, as a generation of social network sites such as Facebook and News Corp's Myspace have become popular among Web users.

"We have decided to discontinue the process of allowing new customers to sign up for GeoCities accounts as we focus on helping our customers explore and build new relationships online in other ways," Yahoo said in a statement. "As part of Yahoo's ongoing effort to build products and services that deliver the best possible experiences for consumers and results for advertisers, we are increasing investment in some areas while scaling back in others." Shares of Yahoo were up a penny to $14.49 in afternoon trading on the Nasdaq.

Actor Feroz Khan passes away

2009-04-27T12:57:00.000+05:30

Bollywood actor Feroz Khan on Monday lost his battle with cancer and passed away at his farmhouse in Bangalore.

The 69-year-old actor-filmmaker died at 1:30 am at his farmhouse, family sources said. Khan is survived by actor-son Fardeen Khan and daughter Laila Khan.

He was diagnosed with cancer and was being treated at a Mumbai hospital for a long time. His funeral will take place later in the day in Bangalore, sources said.

The actor was born and brought up in Bangalore. The Bollywood actor of yesteryear was known for his hit films of the 70s and 80s which include 'Dharmatama', 'Qurbani' and 'Jaanbaz'.

The latest film he featured in was the blockbuster hit Welcome in 2007.

SP rules out alliance with Third front or Left

2009-04-26T19:03:00.000+05:30

Ruling out any post-poll tie-up with the Third Front or the Left parties as long as the Bahujan Samaj Party is with them, the Samajwadi Party on Sunday said it was, however, not averse to an alliance with the Congress.

"As long as our arch rival BSP chief Mayawati is with the Left parties or the Third Front, we are not going for any post-poll alliance them," SP general secretary Amar Singh told media-persons in Kolkata on Sunday. However, the party is not averse to an alliance with the Congress, he said.

"The Congress-led United Progressive Alliance ran the government for the last one year with our help after the Left parties withdrew support on the signing of the civil nuclear deal but they turned down our offer of 17 seats in Uttar Pradesh. We had never been so ungrateful...however, we are not averse to a tie-up with the Congress in post-poll scenario if the situation demand," Singh said.

Launching a scathing attack against the Communist Party of India-Marxist leadership, Singh said, "We had good relation with the CPM during the time of Harkishen Singh Surjeet, but Prakash Karat is altogether a different person." On the question of prime ministership, he said, "The emerging situation will decide who will be the prime minister. Let the Congress first win the confidence of magic number."

10 don'ts for smart stock market investing

2009-04-26T18:54:00.000+05:30

This is a great check list of 10 habits, impulses and tendencies you steer clear of in order to keep your investments healthy.

1. Don't be arrogant

The market teaches humility and that is how you must approach it. As soon as you believe you know why the market acts the way it does, you will be proven wrong. Arrogance can kill a portfolio. You must be able to admit defeat and preserve enough capital to fight again.

Following point and figure charts, which depict the battle between supply and demand, helps keep you out of the 'I know why' attitude of investing.

2. Don't wait until you feel comfortable to buy when a sector reverses up

Falling into the waiting trap is a great way to ensure that you buy the stock at a higher price. When sectors reverse up from oversold levels, it is often when the news is the most dire.

Conventional wisdom would suggest this is the last place in the world you would want to invest. Buying at this time is gut wrenching, but to be successful you must act with complete confidence.

As the sector moves higher, the comfort level increases. If you use comfort level as your guidance, however, you will for sure leave a lot of money on the table, or worse, buy as the sector peaks.

3. Don't be afraid to buy strong stocks

Don't avoid stocks just because they have gone up. Doing so will keep you out of the long-term winners. In the United States, for example, this mentality would have kept you out of General Electric, which was up 188 per cent between January 1995 and December 1997 only to see it rally another 96 per cent by the end of 2000. It also would have kept you out of Cisco, which was up 376 per cent between January 1995 and December 1997, and then it moved up another 312 per cent by the end of 2000. These are only two examples, but there are many others.

More important than how much the stock is up is its supply and demand relationship. By evaluating the point and figure chart, you can gain insight into this relationship and whether or not the stock is likely to move higher. Stocks that double can easily double again. Don't miss out on these great opportunities.

4. Don't sell a stock simply because it has gone up

Doing this cuts profits short. Buying a stock right is only half the battle. You have to be able to sell it right to win the war. Just because a stock has rallied 30 per cent or 50 per cent, don't be tempted to take your trade off for that reason alone.

Consider trimming the position and leave part on the table to continue in the uptrend. Let profits run.

5. Don't buy stocks in extended sectors because 'it's different this time'

On the surface, the stock market appears different all the time. The leadership changes: in come new stocks into the Nifty 50, and then out they go. Small-cap stocks outperform for a while, then it's back to the large caps.

However, the underlying forces that drive the stock market are always the same. They are true and time-tested and do not change. They are supply and demand. That's why buying sectors that are extended (overbought) will not be different this time.

6. Don't try to bottom fish a stock in a downtrend

'The trend is your friend' is a true statement. So don't go against it without some inkling that the trend has changed.

Bottom fishing a stock in a downtrend is the opposite of being afraid to buy strong stocks. Do not buy a stock just because it fell sharply. You want to buy a stock that is likely to move higher, not one that is not likely to fall further.

At a minimum, wait for the stock to show a sign that demand is back in control and suggesting higher prices. That may be a simple buy signal on the chart or a reversal back to the upside after holding an area of support. Also remember why you initiated the position. Be careful not to let a trade turn into something else.

7. Don't buy a stock simply because it is a 'good value'

These days, value is in the eyes of the holder, and therefore it is a subjective term at best. If a stock has become a good value, ask why. This is important, because a stock can stay a good value by not moving for the next decade, or worse, become a better value by dropping another 20 per cent.

The true value of a stock is determined by its capital appreciation potential, not numbers on a balance sheet. The basis for capital appreciation lies in the supply and demand relationship of the stock. Appreciation can occur only if demand grows stronger for the stock and buyers are willing to pay a higher price. Watch the point and figure charts to determine if a stock is likely to move higher in price and become a good value.

8. Don't hold on to losing stocks and hope they come back

Hope is eternal, but your portfolio is not. Holding on to a losing stock is the best way to let your losses run. Combine this mistake with selling a stock that has gone up and you can create a portfolio of dogs.

When buying stocks, there will always be some losers: Count on it. However, how you manage that loss often determines the success or failure of the overall portfolio. Keep losses small so that you have the capital to play again. Hanging on to losing positions, hoping that they will come back, can be deadly.

A $50 stock that is stopped out at $40 is a 20 per cent loss. It's a bad trade, but it is manageable. In order to recoup that loss you would have to make 25 per cent on a $40 stock. What if you held on to that $50 stock, hoping that strong earnings would come in and turn it around, but instead it continued lower to $25?

Finally, you decide to exit, but now it takes a 100 per cent return from a $25 stock just to get back to even. Those results are hard to find, and if you are able to find one, you don't want to waste it on getting back to even

Learn to recognize your losing positions for what they are. If a stock cannot trade above its support line or is not outperforming the averages, find one that is and swap it.

9. Don't pursue perfection

There are two types of mistakes to discuss here. The first is the constant belief that there is a better system out there, and you need to find it.

Using a new system to invest each week will not get you to your goal. You will become good at nothing and moderate to bad at everything. To be good requires that you stay focused, disciplined, and skilled at whatever methodology you choose.

You need to have the strength of conviction in your chosen discipline to learn from mistakes rather than to run away from them and find another methodology. There is no Holy Grail in investing.

The second mistake is to wait for the perfect trade. There is no such thing. If you only buy stocks that have all positive attributes you will maintain a portfolio of cash. Rarely, if ever, do you find a stock that has all the pluses on its side.

Look for the big ones like relative strength, trend, and signal. Also remember that 80 per cent of the cause of price movement in a stock is based on the market and sector. You are better off being approximately right than precisely wrong.

10. Don't do anything based on a magazine cover

Following the hot news that appears on magazine covers is a shortcut to the poor-house. Why should you follow the advice of someone who has just moved from the society pages to the business section?

Feudalism responsible for Satyam fiasco: N R Narayana Murthy

2009-04-20T16:41:00.000+05:30

Technology services titan and founder chairman of Infosys Technologies N R Narayana Murthy has had a deep impact on the

$60 billion IT industry which grew on the back of global sourcing of services. Over the years Mr Murthy has delivered over a 100 lectures on various fora spanning the world. Some of those select lectures on topics ranging from globalization, corporate governance, leadership and entrepreneurship are now part of the just published book: A Better India A Better World. In a 72-minute interview with ET at the company's guest house in South Delhi, Mr Murthy talks about the book, the political scene, the Satyam episode and its impact on India Inc and what the country now needs and more. Excerpts from the interview:

What all have you touched upon in this book and how are the lectures relevant now?

While the country has made considerable progress in the last 18 years after the first wave of economic reforms we are still lagging. We have one of the highest GDP growths in the world, our exports are much larger than what they were 10 years ago, our software industry has brought tremendous laurels to India, we have done well in technology industry, production and even in sports.

Yet we have been unable to redeem the pledge that our founding fathers took when India got independence. That is to provide decent access to education, nutrition, healthcare and welfare to the poorest of the poor. India has the largest mass of illiterates in the world, largest mass of poor people in the world, 250 million people don't have access to safe drinking water, 650 million people do not have access to decent sanitation. So this whole paradigm of 8-9% GDP growth becomes somewhat irrelevant when you look at these aspects.

So the real challenge we have is to bring inclusive growth. In order to bring inclusive growth in a country like India we need to solve three pieces of development -- first is creating a public opinion that values good work ethic, honesty, discipline, secularism. Second, we need to develop a cadre of leadership who espouse these values and serve as role models and demonstrate leadership by example. Third, we need the determination of the elite and the powerful in the society to eschew creation of asymmetry of benefits in their favour vis-à-vis the common man. Only when these three conditions are fulfilled will we be able to create a fair, just, equitable and inclusive growth in our society. In essence this is the fundamental thesis on which the book is based.

I have all along said that people practicing values, leaders leading by example and the elite and the powerful relating to the realities are absolutely essential for a country and a company to make progress. In some sense, on a very small, simple canvas I have painted a very simple drawing at Infosys and that painting has been reasonably successful.

Based on that experience I have been talking to people, students, in India and abroad, to enlightened citizens on various issues within this framework of values and leadership and those pieces form the basic gist of this book.

India launches spy satellite RISAT

2009-04-20T16:38:00.001+05:30

Aiming to enhance India's defence surveillance capabilities, ISRO on Monday morning launched its first Radar Imaging Satellite (RISAT)

from Sriharikota.

As per specifications, RISAT is different from previous remote sensing satellites as it uses Synthetic Aperture Radar (SAR), equipped with many antennas to receive signals that are processed into high-resolution pictures.

The SAR, developed by Israel Aerospace Industries, gives RISAT defence capabilities.

Cyber-squatting: The new gateway to identity theft

2009-04-20T16:36:00.000+05:30

NEW DELHI: That old scrounge, cyber-squatting, is back with a vengeance especially for a host of global brands in India. With domain name prices

falling and more top-level domains (such as .biz, .cn, .mob and lately .in) getting accredited, cyber squatters are back in business full time.

Global brands across categories –– Monster Jobs, PepsiCo, SonyEricsson, Siemens, McAfee or search giant Google — have of late been at the receiving end of squatters. Take for instance, world’s largest job site Monster.com which was squatted in the name of one Usha Rani, a Hyderbad-based squatter which registered Monster.in and Monsterjobs.in.

The global jobsite filed a case with the .in registry saying that “the respondent had registered the domain in bad faith which impugned its brand.” After a resolution process, the domains were taken away and allotted to Monster India. “We have a fast track dispute resolution process where by decisions are transferred within 30 days of filing a complaint,” says a National Internet Exchange of India (NIXI) official, which handles the .in registry.

With little disincentive for squatting under Indian cyber laws, even Pepsico.in, Siemens.in and Sonyeriscsson.in were squatted recently. “There is no provision in the current or proposed IT act in India to punish cyber-squatters. At best, the domain name can be take back,” says cyber lawyer Pavan Duggal, who has dealt with several squatting cases.

With domain name prices falling to as low as Rs 200, squatters seem to be on a domain binging spree. Apart from global brands, celebrities domains are also targeted by squatters. For instance, AmitabhBachchan.in is currently undergoing an auction on the internet, with a minimum price bid of $1,000. Similarly, Soniagandhi.com is registered by Indianmagazine.com a news and views website. Indian film actor Gul Panag is also facing tough time to get her domain back. “I’m not going to pay the squatters $20,000 which they are demanding for GulPanag.com,” she was quoted as saying recently.

Other global brands have also been targeted. Netgear.co.in, Ushaworld.in, Baccarat.in were all squatted and won back recently by respective companies. Indian internet companies whose identities were squatted are Agencyfaqs.com and Rediff.in While Google suffered the squatting of Gmail.in. Mcafee.co.in was also squatted.

One of the world’s largest Flash memory maker Kingston also suffered with its Kingston.in and Kingston.co.in being squatted in India. Though there is no legal compensation under IT Act, .in registry has taken proactive steps to grant compensation to victim companies to deter squatters from further stealing domains. Most squatters however operate under guise of obscure names.

Pirate Bay founders found guilty

2009-04-18T09:51:00.000+05:30

"Frederik Neij, Gottfrid Svartholm Warg, Carl Lundstrom and Peter Sunde were found guilty of breaking copyright law and were sentenced to a year in jail.
"They were also ordered to pay $4.5m (£3m) in damages.

"Record companies welcomed the verdict but the men are to appeal and Sunde said they would refuse to pay the fine.

"Speaking at an online press conference, he described the verdict as "bizarre.""

New Linux rootkit technique presented

2009-04-18T09:40:00.000+05:30

Anthony Lineberry, a Linux expert, announced during his presentation, "Alice in User-Land: Hijacking the Linux Kernel via /dev/mem", at the Black Hat security conference now taking place in Amsterdam that he will shortly be publishing the libmemrk library. He says Libmemrk works in both 32-bit and 64-bit environments.
This offers rootkit developers a new way to hide files or processes, or interfere with network traffic. The trick is that, without requiring extensive rights, libmemrk uses the /dev/mem device driver to write arbitrary code from userspace into main memory. /dev/mem is an interface that enables use of the physically addressable memory. For example XServer and DOSEmu, both use it. Lineberry says introducing rootkits via /dev/mem is also less obvious than the established route via loadable kernel modules (LKMs).

The library relieves a rootkit programmer of the "laborious" work of translating virtual memory addresses to physical ones and identifying a memory range that can be exploited for the attack. An attacker can't overwrite the existing system calls and replace them with his own code until the suitable ranges, normally used by the kernel, have been located. The real contents written into memory by the kernel are meanwhile shifted into a buffer.

The detailed steps required for a successful attack, which are handled by libmemrk, are described by Lineberry in his white paper, Malicious Code Injection via /dev/mem. However, Lineberry states there that an attack fails in virtual environments because the hypervisor behaves differently from unvirtualised hardware. Lineberry asks his audience to bear in mind that, regardless of libmemrk, the whole attack must be hand-programmed in assembly language. In future, Lineberry intends that libcc will be used in order to at least reduce the impact of this hurdle.

Lineberry also gives some tips on how the Linux world can protect itself against rootkits of this kind. He believes it should be enough to modify the memory driver so that it doesn't allow the write/read pointer lseek to look for more than 16 kilobytes in the memory area. Current versions of Red Hat and Fedora are inherently secure, because their kernel already incorporates the features of SELinux (Security Enhanced Linux).
Lineberry says there are also corresponding improvements in version 2.6.26 of the mainline kernel. For that purpose, the kernel was given two new functions: range_is_allowed() and devmem_is_allowed(). But this protection, he says, won't be effective unless the preprocessor directive CONFIG_STRICT_DEVMEM has been enabled when the kernel is compiled. Otherwise, range_is_allowed() always gives returns success. Lineberry says that the kernel configuration setting STRICT_DEVMEM, which sets CONFIG_STRICT_DEVMEM, is not activated by default during kernel compilation. He was unable to say when libmemrk would be available for downloading, as he was still engaged in eliminating its last weaknesses.

The technique of using the /dev/mem interface is not totally new. An article on Linux on-the-fly kernel patching without LKM appeared in Phrack back in 2001, describing a similar method using /dev/kmem/. The authors were already thinking about possible uses for /dev/mem then, but didn't go on to check them out.

3 Minutes to 3 Terabytes: VIA ARTiGO and FreeNAS Store Terabytes in a Shoebox

2009-04-18T09:39:00.001+05:30

It truly is a beautiful thing when something just works. This is especially true with computers. High capacity storage has become almost a commodity with the price of an external 1 TB USB hovering around $100. All you have to do is plug in the power and connect the USB cable, and you've got instant storage expansion. Works great for a single computer, and you could even unplug it from one and plug it into another. While that does work, it tends to get old after a while, and if you're using that method for backups, you will more than likely end up forgetting or just quitting altogether at some point.

Network Attached Storage (NAS) is one answer to sharing large storage devices over a network. You could buy a NAS device from your favorite local or Internet supplier, but chances are you'll wind up with something less than what a "real" NAS has to offer. That's where VIA's ARTiGO A2000 comes in. The A2000 fits a full-featured computer with space for two 3.5" SATA drives in a package about the size of a shoebox. Add to the hardware the FreeNAS open source software and you've got a really capable storage solution.

Installation

Assembling the hardware couldn't be simpler. The ARTiGO A2000 has one screw on the rear of the box holding the case shell in place. Once that's out you can remove the front clip-on cover with two fingers. That exposes two slots for SATA hard drives and the CompactFlash slot. The hard drives slide easily on the rails and plug right into a connector at the back of the drive bay. Tighten two screws on each side to secure the drive to the frame, and you're all done. We used two Seagate Barracuda 7200.11 1.5 TB drives for this review.

Getting the software up and running is a little more involved but not much. Step one is to download the FreeNAS software image from the project download page. The one you want is the latest FreeNAS Image file. For the VIA ARTiGO you'll need the i386 version. While you're on the download page you can also get a copy of the Quick Start Guide and User Manual. Once you have the image downloaded you must write the image to a CompactFlash (CF) device. The image itself is a little under 30 MB, so you could use an old 64 MB device if you happen to have one lying around.

Writing the image to the CF disk requires root privileges. The trick is to know that the .img file is actually compressed, requiring a gunzip command before writing to the disk. You can do it with one command as follows:

root@anand-ubuntu:~/ gunzip -c FreeNAS-i386-embedded-0.69.4276.img | sudo dd of=/dev/sdc

The key here is the name of your CF device. In our case we used a small USB multi-card reader which assigns a unique device name for all the different slots. The CF slot shows up as /dev/sdc. With that piece of information you're ready to blast your image using the command line string above. You can also use the dmesg command to check yourself after inserting the CF card as in:

dmesg | tail -n 24

That will show you the last 24 kernel messages and should contain a few lines that indicate the device name of the card you just attached.

Infosys cuts variable pay

2009-04-16T18:36:00.000+05:30

Infosys has cut the variable pay for employees, with the cuts being deeper at the higher levels. For senior executives, the variable pa
y, which constitutes nearly 50 per cent of their total compensation, has been slashed by 58 per cent.

“Some boardroom executives have even taken a 70 per cent to 85 per cent variable pay cut,” said T V Mohandas Pai, head of HR in Infosys. The company's hiring has been steadily declining.

In Q4, it added (net) only 1,772 people, compared to 2,772 in Q3, and 2,586 in Q4 of 2007-08. In the whole of 2008-09, it hired (net) 13,663 people, down from 18,946 in the year before.

The company maintained that it will not seek to trim its payroll by laying off software professionals. There will, however, be no salary hike this year, as the company plans to keep its operational costs under control.

“We are not laying off anybody and there are no such plans,” said Infosys Technologies HR-director, TV Mohandas Pai.

Infosys plans to hire 18,000 professionals in the current fiscal, including almost 16,000 fresh graduates and experienced hires. It will also recruit around 1,000 non-Indians outside the country to increase the number of foreign professionals in its workforce.

India exercises its ballot, amid bullets and blasts

2009-04-16T18:31:00.001+05:30

Round one of India's Election 2009 ended Thursday with millions queuing up in 17 states and union territories to vote in a new government
amid Maoist violence that marred the democratic exercise in several places and left at least 17 dead.

The world's largest democratic exercise began early at 7 a.m. and ended at most places at 5 pm. A majority of the states reported moderate turnout despite the Maoist attacks on polling centres and security personnel.

Indians voted in 124 constituencies to pick a new 545-seat Lok Sabha in the first step of a four-phased exercise that is widely expected to throw up a split verdict. The Maoist violence affected a dozen constituencies.

About 143 million of India's 714 million voters were eligible to exercise their franchise in the first of five rounds in 185,552 polling centres. A total of 1,715 candidates were in the fray, with over 300,000 electronic voting machines used.

Tens of thousands of election staff and security personnel kept vigil as people voted in all the constituencies of Kerala, Arunachal Pradesh, Meghalaya, Mizoram, Nagaland, Chhattisgarh, Andaman and Nicobar Island and Lakshwadeep. Andhra Pradesh, Assam, Bihar, Uttar Pradesh, Jammu and Kashmir, Maharashtra, Manipur, Orissa and Jharkhand saw partial voting.

But it was a bloody start to the ambitious exercise with Bihar, Chhattisgarh, Orissa, Jharkhand and Maharashtra seeing violence and intimidation as Maoist guerrillas tried to implement their election boycott through the barrel of the gun.

At least 17 people were killed as cadres of the outlawed Communist Party of India-Maoist, which seeks to carry out an agrarian-based revolution, targeted polling officials and security personnel across the insurgency hit states.

Landmine blasts in Chhattisgarh and Jharkhand saw 14 people getting killed. In Jharkhand's Latehar area, seven Border Security Force (BSF) personnel and two others heading to an election centre were killed when their bus was blown up.

In neighbouring Chhattisgarh, five polling officials died when Maoists detonated a landmine in Rajnandgaon district. A paramilitary trooper was shot dead in an exchange of bullets in the Maoist stronghold Dantewada.

In adjoining Bihar, a policeman and a Home Guard were killed when over a dozen Maoists opened fire at a polling station in Gaya district.

Reports of gun battles, booths being raided, voters being attacked and electronic voting machines being torched came in from several places in the affected states.

But voters braved it all at the end of the day.

In Andhra Pradesh, once a Maoist bastion, there was 60-65 percent voter turnout, 65 percent in the three parliamentary constituencies of Assam and 60-70 percent in Mizoram, Nagaland, Meghalaya and Arunachal Pradesh, also in the northeast. Manipur was lower at 40 to 50 percent.

In violence-hit Chhattisgarh, Jharkhand and Bihar, about 50 percent of the voters turned to cast their franchise.

Long queues were seen outside booths in Kerala with officials estimating that the turnout would be about 60 percent. It was highest in Ernakulam with 67.5 percent and surprisingly low in Thiruvananthapuram, where the Congress fielded star debutant, former UN under-secretary general Shashi Tharoor, with 45.4 percent.

With 124 of 543 seats going to the polls, the day was decisive for the main political parties battling for power with several key leaders in the fray.

One of them was Railway Minister and Rashtriya Janata Dal (RJD) chief Lalu Prasad, who accused the rivals of intimidating voters in his constituency in Saran in Bihar.

Cabinet minister and Nationalist Congress Party (NCP) leader Civil Aviation Minister Praful Patel said in his constituency Bhandara-Gondia in Maharashtra: "We are with the Congress as alliance partners. That does not stop us from having good relations with the Left. In fact, we may need them later."

Finally, Thursday was all about the struggle to reach the coveted seat of power in Delhi.

As defence minister and Congress leader A.K. Antony said: "Gone are the days of a single party ruling the country. Hence, even though there are secular parties contesting against us in states, when it comes to forming a government in Delhi we will seek the support of all secular parties."

And this is how BJP general secretary Arun Jaitley put it: "The way the campaign progressed and looking at the voting trends in all the states where we are in strength, we are going to improve our performance."

The exercise was also about the people and their determination to vote, no matter what.

Like Kamla Devi, 102, who came to vote at Gurah Brahamana in Jammu region. It was the 20th time she was doing so and the canny voter refused to disclose who she had supported.

But all will be told on May 16 when the millions of votes will be counted.

Sonia calls Advani 'a slave of RSS'

2009-04-15T16:53:00.000+05:30

Describing the Bharatiya Janata Party's [Images] prime ministerial candidate L K Advani [Images] as a 'slave of the Rashtriya Swayamsevak Sangh', Congress president Sonia Gandhi [Images] on Wednesday said the saffron leader could not take any decision 'without the fear of the RSS'.

"It is Advani who is a slave of the RSS because he wants to protect his chair. This is the reason why he panders to each and every wish of the RSS," Gandhi told an election rally in Bidar, Karnataka.

Dismissing Advani's oft-repeated comment about Manmohan Singh [Images] being a weak prime minister, she asked whether the senior BJP leader could take any decision on his own 'without the fear of the RSS'.

Gandhi said that when Advani was the BJP president, he had given a 'secular certificate' to Jinnah during his visit to Pakistan, which created a furore in the RSS and ultimately led to his resignation from the post.

"Advani must remember that a prime minister does not belong to one party but to the whole nation and an insult to him is an insult to the whole country," she said.

"Now the choice is left to the people. They have to decide who is weak, whether it is Prime Minister Manmohan Singh or BJP leader L K Advani," Gandhi said.

Accusing the BJP of 'being sympathetic towards terrorists', Gandhi said it was during the National Democratic Alliance's rule that 'terrorists were released and escorted to Kandahar'.

She said it was the "efficient and diplomatic tackling of the Mumbai terror attacks [Images] by the United Progressive Alliance government which compelled Pakistan to concede responsibility in the issue".

"We will not tolerate any such attack against India," Gandhi said.

Attacking the BJP government in Karnataka for 'improper utilisation' of central funds allotted for development work, she said "we gave crores of Rupees to Karnataka. We really do not know how the funds have been utilised".

She also came down on the saffron party for the 'rampant corruption' in the state and appealed to the people to cast their votes judiciously.

Highlighting the achievements of the Congress government since Independence, Gandhi said whether it is industrialisation or modernisation of the country, nationalisation of banks, empowerment of women through Panchayat Raj institutions or signing the nuclear agreement, it is the Congress which has always taken the lead.

"The people have to take all this into account and make an informed choice," she said.

Build a centralized log management and monitoring system

2009-04-15T16:51:00.000+05:30

Seasoned system administrators know that routinely reading system logs is an important task, but reading endless lines from logs is both time-consuming and boring, especially if you are responsible for a large number of busy servers. In this article I will show you how to set up a system that gathers and archives system logs from many network hosts and emails only important or irregular system events to administrators.
The majority of GNU/Linux distributions uses the good old syslogd system logger by default, which is based on the original 4.3BSD syslogd daemon. Syslogd is a fine system logger, but it lacks some advanced features modern alternatives offer. We will use syslog-ng instead, which provides all the functionality of the traditional syslogd along with some nice enhancements. Among others, it provides powerful filtering capabilities based on message content, and can also be used in a firewalled environment without problems.

Installation is a breeze since most distributions provide binary packages. If you prefer to manually build the program, check the INSTALL file included in the source tarball, which outlines all the necessary steps. Make sure to uninstall syslogd before installing syslog-ng.

The syntax of the configuration file might seem peculiar and complex compared to the traditional syslog.conf syntax, but it offers almost limitless customization options. Make sure to read the syslog-ng.conf man page for information on how to use it.

Since our logging system will gather logs from other hosts, we need to instruct it to listen for network connections. Syslog-ng supports both the TCP and UDP protocols. IANA has assigned the 514/udp port to the syslog service, so we will use that port for maximum compatibility with syslogd and network devices such as routers. If you use syslog-ng on all your hosts, it's better to use the TCP protocol, which is more reliable and firewall-friendly.

Add the following lines to your /etc/syslog-ng/syslog-ng.conf to the appropriate sections indicated by comments (lines starting with #) to enable listening for network connections on a specific IP address, and to archive logs from remote hosts as /var/log/$HOST/$FACILITY (e.g. /var/log/mailserver/mail):

## add this to the options section
create_dirs(yes);
long_hostnames(off);
keep_hostname(yes);

# uncomment the following line only on a LAN with working DNS
#use_dns(yes);

## add this to the source section
source s_udp {
udp ( ip(192.168.1.2) ); # replace with your system's IP address
};

## add this to the destination section
destination df_udp {
file ("/var/log/$HOST/$FACILITY");
};

## add this to the log section
log {
source(s_udp);
destination (df_udp);
};
On the remote hosts add the following lines in /etc/syslog-ng/syslog-ng.conf or /etc/syslog.conf, depending on whether they run syslog-ng or syslogd respectively:

## /etc/syslog-ng/syslog-ng.conf

## add this to the destination section
destination remote_udp { udp("192.168.1.2"); }; # replace with your log server's IP address

## add this to the log section
log { source(src); destination(remote_udp); };

## /etc/syslog.conf

# use tabs instead of space
*.* @192.168.1.2 # replace with your log server's IP address
You can add additional filters to better suit your needs or even log to a database like PostgreSQL. Note that syslog traffic between hosts is unencrypted; if you want to gather logs from remote hosts over the Internet, create SSH tunnels first, for security.

Logcheck

At this point you have configured a full-featured syslog server that gathers and archives logs from multiple servers, but so far you still have to read those logs manually. Now we'll add logcheck to the equation.

Logcheck is an excellent program that parses log files, filters out expected, normal events based on pre-defined regular expressions, then summarizes the remaining entries and emails them to the system administrator's account. Logcheck was previously part of the Sentry tools suite, but since it had been unmaintained for a long time, it was forked by Debian developers, who have done a wonderful job integrating logcheck into the system. Most network daemon packages include logcheck rules out of the box.

To install logcheck on a Debian-based system simply apt-get install the logcheck, logtail, and logcheck-database packages; the last of the three provides lots of ready-made rules for various system events. To install it on other distributions, download the source tarball and read the INSTALL file. Since it is just a shell script it does not need any compilation.

Configuration is simple. First enter which log files you want to be checked by logcheck in /etc/logcheck/logcheck.logfiles. Logcheck supports three levels of filtering: paranoid, server, and workstation. Each level uses a directory named /etc/logcheck/ignore.d.level_name that includes filtering rules files with different verbosity levels. Paranoid produces highly verbose output and should be used only on high-security systems running a minimum number of services. Server should be fine for most systems and is used by default. Workstation filters out most of the messages and thus produces the least verbose output of the three. You can define which filtering level should be used in /etc/logcheck/logcheck.conf, along with other parameters, such as the recipient email address and the subject of the emails.

Logcheck uses standard regular expressions to filter logs. It's not difficult to write custom rules; read the WRITING RULES paragraph of the docs/README.logcheck-database file in the source tarball (or /usr/share/doc/logcheck-database/README.logcheck-database.gz in Debian) for more information. If you're new to regular expressions, this guide might be useful. As an example, check this filter file for Dovecot:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap-login: Login: [.[:alnum:]-]+ \[[0-9.]+\]$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap-login: Disconnected \[[0-9.]+\]$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap$[^[:space:]]+$: File isn't in mbox format: [^[:space:]]+$
Logcheck runs periodically from cron. The default cron job installed by the Debian package (/etc/cron.d/logcheck) runs logcheck every hour or when the system reboots. Unless you want 24 logcheck email messages per day, you should adjust how often it should run by editing the file. I prefer to run it on a daily basis.

EC order gags media on poll eve coverage

2009-04-15T16:49:00.001+05:30

THE Election Commission said on Tuesday that the electronic media cannot telecast anything related to elections which can influence voters in areas where polls are to take place, 48 hours preceding voting.

The Election Commissioner said, TV channels were barred from telecasting poll-related programmes like interviews, discussions and poll analyses and surveys that could influence voters 48-hour before polling day.

In a separate order issued under section 126 of the Representation of People's Act that prohibits displaying any election matter on television or any related medium 48 hours before poll, EC has also banned dissemination of results of opinion and exit polls by the media. The gag on electronic media is seen as unsuitable for multi-phase elections as well as innocent of the ways the media functions.

The official said, "the rule applies even to national channels. During this period, national channels are not allowed to telecast such programmes about Andhra Pradesh. Even animation programmes that could influence voters should not be screened."

Mr Rao explained that with the first phase polling scheduled for April 16, TV channels would not even be allowed to air poll-related analyses of seats going to the polls on April 23. Failure to comply would mean imprisonment up to two years or fine or both.

SC seeks undertaking from Varun, hearing put off

2009-04-14T15:17:00.002+05:30

THE Supreme Court has adjourned the hearing on Varun Gandhi’s bail plea till Thursday, which means the BJP leader will have to spend a few more days in jail.

The apex court said that Varun can be free provided he gives in writing to the court that he will make no provocative speeches during his campaign. Varun's lawyer told the court that he would give the undertaking.

Now, the SC has asked the Uttar Pradesh government if Varun gives an undertaking, will it be acceptable to the UP government.

Varun Gandhi, who has been detained in the Etah jail, has challenged his detention under the National Security Act (NSA) by the Uttar Pradesh government.

The apex court bench headed by Chief Justice KG Balakrishnan will consider the plea for Varun's release to enable him to file his nomination papers as well as to contest elections from Pilibhit. The 29-year old was arrested under section 153 A of the Indian Penal Code for delivering a communal speech during an election rally in March The NSA was slapped on him following violence in Pilbhit during his surrender before the court.

While Varun's lawyers are likely to argue that the CD containing the hate speech is doctored, and therefore inadmissible as material evidence to book him under the NSA, the Uttar Pradesh government is expected to take a stand that Varun’s release would pose a threat to peace and could lead to a serious law and order problem in the state.

In its affidavit, the Mayawati government has described Varun as a "national threat to communal peace and harmony", thereby defending its action of invoking the NSA.

Laloo-Mulayam-Paswan claim to be kingmakers

2009-04-14T15:17:00.001+05:30

THE strong political formation among foes-turned-friends, Laloo, Mulayam and Ram Vilas Paswan are almost confident that they would be able to have a significant say in the government formation process at the Centre after the elections.

Speaking at its first show as the 'Fourth Front' in Uttar Pradesh, the three leaders addressed a joint rally at Saifai in Etawah district of Uttar Pradesh on Thursday, claiming that the next government at the Centre cannot be formed without them.

The troika, comprising RJD, LJP and SP, was formed in March 2009, and will contest 120 seats in the three states. The leaders repeatedly said that the new coalition is part of the UPA, and will work together to stop communal forces from coming to power.

In the rally, Laloo Prasad Yadav said,"Three brothers have come together, not only to win Lok Sabha elections, but also to fight communalism. We will show our strength in the cow belt."

The three joined hands after the Congress said it would not have a nationwide alliance. After Saifai, they are planning to hold joint rallies in Varanasi, Lucknow and several places in Bihar.

Indian Elections - Varun Gandhi's Hate Speech

2009-04-14T15:16:00.001+05:30

[Note to non-Indian visitors and unacquainted Indians: Varun Gandhi, 29, is the nephew of Sonia Gandhi and grandson of Indira Gandhi, former Indian Prime Minister. However, unlike Sonia and Indira who were in the Indian National Congress Party, Varun is in the Hindu nationalist Bharatiya Janta Party (translated into the Indian People’s Party). This is the first time he is running for office. BJP is not new to inciting religious violence against Muslims (to be fair, some Indian Muslim groups are often involved in acts that give the BJP ‘excuses’ to do what they do. However, the BJP and its sister and mother organizations, the Bajrang Dal, Vishva Hindu Parishad and the RSS, are no less and they are collectively ‘guilty’ at times). In India, Hindu and Muslim fundamentalism feed each other. A regressive Islamic philosophy has met a bigoted Hindu Nationalist one. The BJP derives its cadres from those who believe the Congress “appeases” the minorities (read, “Muslims”) at the expense of “Hindu interests”.]

Here are portions of the speech made by Varun Gandhi in Pilibhit amid a crowd that was reeling in anti-Muslim hatred due to certain incidents in recent local history, as appeared in the Indian Express, March 18 ’09:

I would add here that if the people of that region had really seen outrages by the Muslim community, the campaigning leaders should have spoken about bringing law-and-order and justice to the region. To call for the heads of Muslims is nothing short of barbaric.

After a CD was released of this speech by some private entities, presumably political rivals, Varun denied making any of those statements claiming that the speech was doctored and the voice isn’t his. On TV today, he gave an example of the doctoring. He said he never referred to Muslims as “Katuas” (derogatory term for a Muslim referring to their circumcision – equivalent to calling an African American as ‘nigger’) and instead was referring to “vote Katuas” which he claims means those non-serious political candidates who run in elections to “cut votes” of popular candidates. However yesterday, in the Indian Express, he claimed he was referring to “galat tatvas” roughly meaning “anti-social elements”, and not “Katuas”. He claimed that was added later. Two explanations in two days. And to top that, claims that he “never spoke those words”.

He seems guilty on the face of it. The video doesn’t seem doctored. However, I would be wrong to call him that until and unless he is proven guilty in a court of law. Innocent unless proven otherwise. But hunch is that he isn’t innocent.

India election 2009 : world's biggest democracy

2009-04-14T15:15:00.000+05:30

The General elections in India are the elections by which the Indian electorate chooses the diverse members of the Lok Sabha in the Parliament for the subsequently term of five years. The voters also ultimately votes for the Prime Minister as the head chosen by the majority party or the majority alliance becomes the next Prime Minister.

The General elections is the prime election work out in the world. With the dawn of EVMs, the election process has become more protected and swift.
Indian general election, 2009

All 543 seats in the Lok Sabha

An election is a administrative process by which a inhabitants chooses an character to hold recognized office. This is the natural device by which present representative democracy fills offices in the legislature, sometimes in the executive and judiciary, and for regional and local government. This course of action is also used in many other privileged and business organizations, from clubs to voluntary associations and corporations.
The worldwide use of elections as a contrivance for selecting legislature in modern democracies is in contrast with the follow in the democratic archetype, ancient Athens. Elections were well thought-out an oligarchic institution and most political offices were onthe top using sortition, also known as allowance, by which officeholders were chosen by lot.
Electoral reform describes the process of introducing pale electoral systems where they are not in position, or humanizing the fairness or usefulness of existing systems. Psephology is the study of results and other statistics relating to elections.

Leader:- Manmohan Singh
Last election:- 145 seats, 26.7%
Leader's seat:- Assam (Rajya Sabha)
Party:- Congress

Leader:- Lal Krishna Advani
Last election:- 138 seats, 22.2%
Leader's seat:- Gandhinagar
Party:- BJP

India will seize general elections to the 15th Lok Sabha in 5 phases on April 16, April 22, April 23, April 30, May 7 and May 13, 2009. The outcome of the election will be announced in single phase on May 16, 2009.

According to the Indian Constitution, elections in India for the Lok Sabha (the national parliament) must be seized at least every five years under conventional circumstances. With the last elections held in 2004, the term of the 14th Lok Sabha expires on June 1, 2009.
The election is conducted by the Election Commission of India, which estimates an electorate of 714 million voters, an increase of 43 million over the 2004 election. During the financial plan presented in February 2009, Rupees 1,120 Crores (Approx. EUR 180 M) was budgeted for election operating expense.

Think Twice before you choose.

India Election '09

2009-04-14T15:14:00.000+05:30

Election campaigning is in full swing in India and amidst all the frenzy, the Pilbhit constituency in Uttar Pradesh, has come under the scanner after one of its young BJP candidates, Varun Gandhi courted controversy over his allegedly communal and inflammatory campaign speeches during election rallies in his constituency on March 6th and 8th, 2009.

Over the next few weeks, video footage of Varun Gandhi's speeches were repeatedly splashed across TV channels. When questioned by the media and the Election Commission. Varun stated that the audio in the footage had been doctored and it was a ploy against him. Unmoved by his denial, the Election Commission sent him a show cause notice for violating the Model Code of Conduct, and later, on 22 March 2009, found him guilty of making ‘hate speeches'.

A spate of criminal cases lodged against him, on 29th March, Varun Gandhi surrendered before a local court in Pilbhit. He was arrested and jailed. The State Government has now booked him under the National Security Act (NSA) on “charges of inciting communal passion by making provocative and inflammatory speeches during [election campaign] meetings”.

As of today, Varun Gandhi continues to be in the eye of the storm as various political parties and their leaders try to gain maximum mileage out of the incident. The Rashtriya Janata Dal supremo Lalu Prasad Yadav has gone as far as to court controversy himself after making a speech berating Varun. The BJP on the other hand, has renewed it's stance of backing it's protege Varun Gandhi with both political and legal aid.

The blogosphere too has been abuzz with opinions on the Varun Gandhi controversy.

Youth ki Awaaz writes:

If young leaders like Varun give such defamatory remarks, what can we expect from the others? The fact that India is a country with communal diversity makes it mandatory for each and every citizen to have a feeling of brotherhood.

Blogger ak too is apprehensive about young candidates like Varun Gandhi, who could also be tomorrow's leaders spewing such rhetoric. He says:

To hear someone so young like Varun Gandhi give out that hate speech against the muslims in such mean tones was just shameful. Are these the young leaders that is going to takle(sic) India forward??

Another blogger Vijay Vikram discusses what he feels was the motivation behind Varun's rhetoric.

It is a sad fact of Indian - for that matter all democratic polity - that aspiring statesmen have to appeal to the lowest common denominator for electoral gains. That is precisely what Varun Gandhi was doing. Varun Gandhi's remarks were borne out of political necessity, nothing else.

On the other hand, some bloggers have come out in support of Varun Gandhi, seeing him as a scapegoat in the drama of India's minority-vote politics. Sreekrishnan Venkatesan writes:

i did not find anything wrong in Varun Gandhi's speech. He warned any other religious fanatics killing Hindus, which to me looked very much normal. In fact this isn't as bad like congress which goes all out to playing the communal card, with non hindu religions. Hypocrisy in all its strength. Supporting a minority religious community is “Secular” while supporting Hindus is “communal”.

In this entire controversy, the role of the MSM has also come under scrutiny. Shahid Siddiqui of Media-Mania wonders why the TV channels devoted as much as over 22.57 hrs of prime time playing back the video footage. He asks:

If the media really believed that Varun Gandhi’s speech would cause unrest among a section of the people, did the repeat telecasts of the speech make any sense?…All the TV channels have overplayed the issue. It was not even authenticated if the CD was original. As per the ethics of journalism, it should not have been played as it has been done, especially during the elections….The role of media is certainly open to question. While reporting that it was a “hate speech” “blatantly communal” etc, did the media behave responsibly by telecasting the tape umpteen times a day for the last few days?

Bloggers are also discussing whether Varun Gandhi should have been booked under the NSA. Many of them seem to echo the words of the Chief Minister of Jammu & Kashmir, Omar Abdullah, who stated that “The hate speech of BJP's Lok Sabha candidate Varun Gandhi did not threaten national security and a law other than the National Security Act could have been invoked to deal with it”. In this context,Vinay writes:

… punishing him under NSA is unwarranted. Even though his speech had the potential to disturb public order, the warning by election Commission and subsequent FIRs under Representation of People Act were sufficient.

The CD containing his speech came into light some 15 days later after he gave it. It means his speech did not lead to any violence immediately, which is actually case in most of the instances.

Listening to his speech one can say that it was more of rhetoric in spite of being venomous.

Nonetheless he deserves punishment, but not under NSA.

He has been arrested under the preventive detention clause of NSA, which is clearly doctored to prevent him from contesting elections.

Varun's lawyers have challenged his detention in the Supreme Court. The case will come up for hearing on April 13th.

Script to Install CentOS 5 on Amazon

2009-04-14T15:13:00.000+05:30

#!/bin/bash -e
# Copyright (c) 2007 RightScale Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#
# Uncomment and edit these for production use
# the full pathname is required for the certificates and
# private keys. Examples are below.
#
#export EC2_CERT=/home/ec2/etc/cert.pem
#export EC2_HOME=/home/ec2
#export EC2_PRIVATE_KEY=/home/ec2/etc/pk.pem
#export AWS_ACCOUNT_NUMBER=
#export AWS_ACCESS_KEY_ID=
#export AWS_SECRET_ACCESS_KEY=
#export AWS_BUCKET=
#export IMAGE_NAME=

echo "Hello $USER, Lets get started installing CentOS 5"

echo "........................................"
showOpts () {
echo "Please Select an Option or 8 to quit"
echo "0) Set EC2 Variables"
echo "1) Create and Mount Image"
echo "2) Installing Yum and CentOS 5 Base"
echo "3) Install Additional Packages"
echo "4) Install RightScale Customizations"
echo "5) Clean Up FileSystem and Bundle Image"
echo "6) Upload Image"
echo "7) Clean Up"
echo "8) Quit"
}
showEC2Opts () {

echo "Please Select an Option or 4 to quit"
echo "1) Set EC2 Variables"
echo "2) Show EC2 Variables"
echo "3) Set AWS Bucket & Image Name"
echo "4) Back"
}

while [ 1 ]
do
showOpts
read CHOICE
case "$CHOICE" in
"0")
while [ 1 ]
do
showEC2Opts
read EC2CHOICE
case "$EC2CHOICE" in
"1")
echo "Warning !!!"
echo "The full pathname is required for the Certificate"
echo "and Private Keys to work properly"
echo " "
echo "Please Enter Your Certificate Path"
read EC2_CERT_PATH
export EC2_CERT=$EC2_CERT_PATH
echo "Please Enter You Private Key Path"
read EC2_PRIVATE_KEY_PATH
export EC2_PRIVATE_KEY=$EC2_PRIVATE_KEY_PATH
echo "Please Enter Your AWS Account Number"
read AWS_ACCOUNT_NUMBER_TEMP
export AWS_ACCOUNT_NUMBER=$AWS_ACCOUNT_NUMBER_TEMP
echo "Please Enter Your AWS Access Key"
read AWS_ACCESS_KEY_ID_TEMP
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_TEMP
echo "Please Enter Your AWS Secret Access Key"
read AWS_SECRET_ACCESS_KEY_TEMP
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_TEMP
echo "Done"

;;
"2")
echo "------------Parameters-----------------------"
echo "EC2 Certificate Path:" $EC2_CERT
echo "EC2 Private Key Path:" $EC2_PRIVATE_KEY
echo "AWS Account Number:" $AWS_ACCOUNT_NUMBER
echo "AWS Access Key:" $AWS_ACCESS_KEY_ID
echo "AWS Secret Access Key:" $AWS_SECRET_ACCESS_KEY
echo "AWS Bucket:" $AWS_BUCKET
echo "Image Name:" $IMAGE_NAME
echo "------------End Parameters-------------------"
echo ""
;;

"3")
echo "Please enter in the AWS Bucket"
read AWS_BUCKET_TEMP
export AWS_BUCKET=$AWS_BUCKET_TEMP
echo "Please Enter Your Image Name ex: myfc6.img"
read IMAGE_NAME_TEMP
export IMAGE_NAME=$IMAGE_NAME_TEMP
showEC2Opts
;;
"4")
break
;;
esac
done
;;
"1")
echo "Creating 10GB Image"
mkdir /mnt/image
dd if=/dev/zero of=/mnt/image/$IMAGE_NAME bs=1M count=10240
echo "Creating File System"
mke2fs -F -j /mnt/image/$IMAGE_NAME
mkdir /mnt/ec2-fs
echo "Mounting File System in /mnt/ec2-fs"
mount -o loop /mnt/image/$IMAGE_NAME /mnt/ec2-fs
mkdir /mnt/ec2-fs/dev
/sbin/MAKEDEV -d /mnt/ec2-fs/dev -x console
/sbin/MAKEDEV -d /mnt/ec2-fs/dev -x null
/sbin/MAKEDEV -d /mnt/ec2-fs/dev -x zero
mkdir /mnt/ec2-fs/proc
mount -t proc none /mnt/ec2-fs/proc
mkdir /mnt/ec2-fs/etc
cat < /mnt/ec2-fs/etc/fstab
/dev/sda1 / ext3 defaults 1 1
/dev/sda2 /mnt ext3 defaults 1 2
/dev/sda3 swap swap defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
EOL
echo "Finished Step 1"
;;
"2")
echo "Installing Yum 3.0"
wget http://linux.duke.edu/projects/yum/download/3.0/yum-3.0.5.tar.gz
tar -xvzf yum-3.0.5.tar.gz
cd yum-3.0.5
make DESTDIR=/ install
echo "Creating Yum Confuration"
mkdir -p /mnt/ec2-fs/sys/block
mkdir -p /mnt/ec2-fs/var/
mkdir -p /mnt/ec2-fs/var/log/
touch /mnt/ec2-fs/var/log/yum.log
cat < /mnt/image/yum.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
reposdir=/dev/null
metadata_expire=1800

[base]
name=CentOS 5 - $basearch - Base
baseurl=http://mirrors.kernel.org/centos/5.0/os/x86_64/
http://mirror.rightscale.com/centos/5/os/x86_64/
enabled=1

[updates-released]
name=CentOS 5 - $basearch - Released Updates
baseurl=http://mirrors.kernel.org/centos/5.0/updates/x86_64/
http://mirror.rightscale.com/centos/5/updates/x86_64/
enabled=1

[extras]
name=CentOS 5 Extras $releasever - $basearch
baseurl=http://mirror.centos.org/centos/5/extras/x86_64/
enabled=1

[epel]
name=Extra Packages for Enterprise Linux 5 - $basearch
baseurl=http://download.fedora.redhat.com/pub/epel/5/x86_64
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=x86_64
failovermethod=priority
enabled=1

EOL
echo "Running Yum"
yum -c /mnt/image/yum.conf --installroot=/mnt/ec2-fs -y groupinstall Base
echo "Finished Step 2"
;;
"3")
echo "Starting Secondary Install"
yum -c /mnt/image/yum.conf --installroot=/mnt/ec2-fs -y install wget mlocate nano logrotate ruby* postfix openssl openssh openssh-askpass openssh-clients openssh-server curl gcc* zip unzip bison flex compat-libstdc++-296 cvs subversion autoconf automake libtool compat-gcc-34-g77 mutt sysstat rpm-build fping rrdtool rrdtool-devel rrdtool-doc rrdtool-perl rrdtool-python rrdtool-tcl vim-common vim-enhanced
yum -c /mnt/image/yum.conf --installroot=/mnt/ec2-fs -y clean packages
cat < /mnt/ec2-fs/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
EOL

cat < /mnt/ec2-fs/etc/sysconfig/network-scripts/ifcfg-eth0
ONBOOT=yes
DEVICE=eth0
BOOTPROTO=dhcp
EOL

cat < > /mnt/ec2-fs/etc/rc.local
touch /var/lock/subsys/local
# Update the EC2 AMI creation tools
echo " + Updating ec2-ami-tools"
curl -o /tmp/ec2-ami-tools.noarch.rpm http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm && \
rpm -Uvh /tmp/ec2-ami-tools.noarch.rpm && \
echo " + Updated ec2-ami-tools"
if [ ! -d /root/.ssh ] ; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-key
if [ $? -eq 0 ] ; then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm /tmp/my-key
fi

EOL
cat < > /mnt/ec2-fs/etc/ssh/sshd_config
UseDNS no
PermitRootLogin without-password
EOL

echo "Finished Step 3"
;;
"4")
echo "Adding RightScale"
mkdir -p /tmp/updates
mkdir -p /mnt/ec2-fs/opt/rightscale/
mkdir -p /mnt/ec2-fs/opt/rightscale/bin
mkdir -p /mnt/ec2-fs/opt/rightscale/etc
mkdir -p /mnt/ec2-fs/opt/rightscale/etc/init.d
mkdir -p /mnt/ec2-fs/opt/rightscale/lib
mkdir -p /mnt/ec2-fs/var/spool/ec2/
mkdir -p /mnt/ec2-fs/var/spool/ec2/meta-data
curl -o /tmp/updates/ec2-ami-tools.noarch.rpm http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
rpm -Uvh /tmp/updates/ec2-ami-tools.noarch.rpm --force --nodeps
#fetch needed packages
echo "Fetch Needed Packages"
curl -o /tmp/updates/linux-2.6.16.33-ec2.tgz http://s3.amazonaws.com/ec2-downloads/linux-2.6.16.33-ec2.tgz
curl -o /tmp/updates/kernel-modules.2.6.16-xenU.tgz http://s3.amazonaws.com/rightscale_software/kernel-modules-2.6.16.33-xenU.tgz
tar -xvzf /tmp/updates/kernel-modules.2.6.16-xenU.tgz -C /mnt/ec2-fs/lib/modules/
#chroot Section
echo "Chroot Time"
mkdir -p /mnt/ec2-fs/tmp/updates
touch /mnt/ec2-fs/etc/mtab
cp -R /tmp/updates/ /mnt/ec2-fs/tmp/
#get rrd-tool
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-devel-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-devel-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-doc-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-doc-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-perl-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-perl-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-php-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-php-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-python-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-python-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-ruby-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-ruby-1.2.23-5.i386.rpm
#curl -o /mnt/ec2-fs/tmp/updates/rrdtool-tcl-1.2.23-5.i386.rpm http://s3.amazonaws.com/rightscale_software/centos/rrdtool-tcl-1.2.23-5.i386.rpm
#get EPEL
curl -o /mnt/ec2-fs/tmp/updates/epel-release-5-2.noarch.rpm http://s3.amazonaws.com/rightscale_scripts/epel-release-5-2.noarch.rpm

cat < <'EOL' > /mnt/ec2-fs/tmp/updates/install-script

echo "starting install"
echo "127.0.0.1 localhost localhost.localdomain" > /etc/hosts
authconfig --enableshadow --useshadow --enablemd5 --updateall
mv /lib/tls /lib/tls.disabled
echo "Disabling TTYs"
perl -p -i -e 's/(.*tty2)/#\1/' /etc/inittab
perl -p -i -e 's/(.*tty3)/#\1/' /etc/inittab
perl -p -i -e 's/(.*tty4)/#\1/' /etc/inittab
perl -p -i -e 's/(.*tty5)/#\1/' /etc/inittab
perl -p -i -e 's/(.*tty6)/#\1/' /etc/inittab
perl -p -i -e 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
perl -p -i -e 's/#ClientAliveInterval 0/ClientAliveInterval 60/' /etc/ssh/sshd_config
perl -p -i -e 's/#ClientAliveCountMax 3/ClientAliveCountMax 240/' /etc/ssh/sshd_config
service network start
echo "Fetching RightScale"
cat < <'SSH' >/etc/init.d/getsshkey
#!/bin/bash
# chkconfig: 4 11 11
# description: This script fetches the ssh key early. \
#

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
[ -r /etc/sysconfig/network ] && . /etc/sysconfig/network

# Check that networking is up.
[ "${NETWORKING}" = "no" ] && exit 1

start() {
if [ ! -d /root/.ssh ] ; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-key
if [ $? -eq 0 ] ; then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm /tmp/my-key
fi
# or fetch public key using the file in the ephemeral store:
if [ -e /mnt/openssh_id.pub ] ; then
cat /mnt/openssh_id.pub >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
fi
}

stop() {
echo "Nothing to do here"
}

restart() {
stop
start
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop}"
exit 1
esac

exit $?

SSH
chmod +x /etc/init.d/getsshkey

rpm -Uvh http://s3.amazonaws.com/rightscale_scripts/syslog-ng-1.6.12-1.x86_64.rpm
curl -o /opt/rightscale_scripts.tgz http://s3.amazonaws.com/rightscale_scripts/rightscale_scripts.tgz
tar -xvzf /opt/rightscale_scripts.tgz -C /opt/
ln /opt/rightscale/etc/init.d/rightscale /etc/init.d/rightscale
chmod +x /opt/rightscale/etc/init.d/rightscale
chmod +x /etc/init.d/rightscale
echo "Modifying Services"
chkconfig --add rightscale
chkconfig --add postfix
chkconfig --add getsshkey
chkconfig --level 4 getsshkey on
chkconfig --level 4 rightscale on
chkconfig --level 4 postfix on
chkconfig --level 4 psacct on
chkconfig --level 4 syslog-ng on
chkconfig --level 4 smartd off
chkconfig --level 4 anacron off
chkconfig --level 4 avahi-daemon off
chkconfig --level 4 avahi-dnsconfd off
chkconfig --level 4 apmd off
chkconfig --level 4 acpid off
chkconfig --level 4 auditd off
chkconfig --level 4 irqbalance off
chkconfig --level 4 mdmpd off
chkconfig --level 4 portmap off
chkconfig --level 4 nfslock off
chkconfig --level 4 syslog off
chkconfig --level 4 sendmail off
chkconfig --level 4 cpuspeed off
chkconfig --level 4 cups off
chkconfig --level 4 autofs off
chkconfig --level 4 bluetooth off
chkconfig --level 4 rpcidmapd off
chkconfig --level 4 rpcsvcgssd off
chkconfig --level 4 rpcgssd off
chkconfig --level 4 pcscd off
chkconfig --level 4 gpm off
chkconfig --level 4 hidd off
chkconfig --level 4 xfs off
chkconfig --level 4 yum-updatesd off
chkconfig --del avahi-daemon
chkconfig --del acpid
chkconfig --del auditd
chkconfig --del irqbalance
chkconfig --del mdmpd
chkconfig --del avahi-dnsconfd
chkconfig --del NetworkManager
chkconfig --del NetworkManagerDispatcher
chkconfig --del dhcdbd
chkconfig --del dund
chkconfig --del firstboot
chkconfig --del irda
chkconfig --del apmd
chkconfig --del smartd
chkconfig --del kudzu
chkconfig --del hidd
chkconfig --del gpm
chkconfig --del pcscd
chkconfig --del bluetooth
chkconfig --del cpuspeed
chkconfig --del cups
chkconfig --del rdisc
chkconfig --del sendmail
chkconfig --del readahead_later
chkconfig --del syslog
chkconfig --del wpa_supplicant
chkconfig --del pand
chkconfig --del netplugd

echo "Fetching Java"
curl -o /tmp/updates/jdk-6u2-linux-amd64.rpm http://s3.amazonaws.com/rightscale_software/jdk-6u2-linux-amd64.rpm
curl -o /tmp/updates/sun-javadb-client-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-client-10.2.2-0.1.i386.rpm
curl -o /tmp/updates/sun-javadb-common-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-common-10.2.2-0.1.i386.rpm
curl -o /tmp/updates/sun-javadb-core-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-core-10.2.2-0.1.i386.rpm
curl -o /tmp/updates/sun-javadb-demo-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-demo-10.2.2-0.1.i386.rpm
curl -o /tmp/updates/sun-javadb-docs-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-docs-10.2.2-0.1.i386.rpm
curl -o /tmp/updates/sun-javadb-javadoc-10.2.2-0.1.i386.rpm http://s3.amazonaws.com/rightscale_software/sun-javadb-javadoc-10.2.2-0.1.i386.rpm
echo "Installing Software"
cd /tmp/updates
curl -o /tmp/updates/bwm-ng-0.5-1.x86_64.rpm http://s3.amazonaws.com/rightscale_software/bwm-ng-0.5-1.x86_64.rpm
rpm -Uvh /tmp/updates/*.rpm --nodeps --force
tar -xvzf linux-2.6.16.33-ec2.tgz
mv linux-2.6.16.33-xenU/ /usr/src/
ln -sf /usr/src/linux-2.6.16.33-xenU/include/acpi /usr/include/acpi
ln -sf /usr/src/linux-2.6.16.33-xenU/include/asm /usr/include/asm
ln -sf /usr/src/linux-2.6.16.33-xenU/include/asm /usr/include/asm-generic
ln -sf /usr/src/linux-2.6.16.33-xenU/include/config /usr/include/config
ln -sf /usr/src/linux-2.6.16.33-xenU/include/keys /usr/include/keys
ln -sf /usr/src/linux-2.6.16.33-xenU/include/linux /usr/include/linux
ln -sf /usr/src/linux-2.6.16.33-xenU/include/math-emu /usr/include/math-emu
ln -sf /usr/src/linux-2.6.16.33-xenU/include/media /usr/include/media
ln -sf /usr/src/linux-2.6.16.33-xenU/include/mtd /usr/include/mtd
ln -sf /usr/src/linux-2.6.16.33-xenU/include/pcmcia /usr/include/pcmcia
ln -sf /usr/src/linux-2.6.16.33-xenU/include/rdma /usr/include/rdma
ln -sf /usr/src/linux-2.6.16.33-xenU/include/rxrpc /usr/include/rxrpc
ln -sf /usr/src/linux-2.6.16.33-xenU/include/sound /usr/include/sound
ln -sf /usr/src/linux-2.6.16.33-xenU/include/video /usr/include/video
ln -sf /usr/src/linux-2.6.16.33-xenU/include/xen /usr/include/xen

echo "Configuring Java Home"
echo "export JAVA_HOME=/usr/java/default" >> /etc/profile.d/java.sh
chmod +x /etc/profile.d/java.sh
echo "Add EC2 Tools"
mkdir /home/ec2
mkdir /home/ec2/etc
curl -o /tmp/ec2-api-tools.zip http://s3.amazonaws.com/rightscale_software/ec2-api-tools.zip
unzip /tmp/ec2-api-tools.zip -d /tmp/
mv /tmp/ec2-api-tools-1.2-13740/* /home/ec2/
ln -sf /usr/lib/site_ruby/aes/ /usr/lib/ruby/site_ruby/1.8/aes
rm -fr /tmp/ec2*

chmod -R o-w /home/ec2

echo "More EC2 Mods"
cat < <'PROMPT'> /etc/profile.d/prompt.sh
PS1="[\u@\h:\w] "
PROMPT
chmod +x /etc/profile.d/prompt.sh
cat < <'EC2'> /etc/profile.d/ec2.sh
export EC2_HOME=/home/ec2
export EC2_CERT=
export EC2_PRIVATE_KEY=
export AWS_ACCOUNT_NUMBER=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PATH=$PATH:/home/ec2/bin/
EC2

chmod +x /etc/profile.d/ec2.sh
ln -f /opt/rightscale/etc/motd /etc/motd
echo "RubyGems"
wget http://rubyforge.org/frs/download.php/20989/rubygems-0.9.4.tgz
tar -xvzf rubygems-0.9.4.tgz
cd rubygems-0.9.4
ruby setup.rb
gem update
gem source -a http://mirror.rightscale.com

#cat < <'GEM'> /root/.gemrc
#gem: --source http://mirror.rightscale.com
#GEM

mkdir -p /tmp/updates
curl -o /tmp/updates/s3sync.gem http://s3.amazonaws.com/rightscale_software/s3sync-1.1.4.gem
gem install /tmp/updates/s3sync.gem
gem install xml-simple net-ssh net-sftp -y
updatedb
cat < /etc/cron.daily/do_amitools_update.sh
#!/bin/bash
#
# do_amitools_update.sh: updates ami-tools to the latest version..
#
## Include Files:
. /var/spool/ec2/meta-data.sh
. /var/spool/ec2/user-data.sh

# Update the EC2 AMI creation tools
echo " + Updating ec2-ami-tools"
curl -o /tmp/ec2-ami-tools.noarch.rpm http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm && \
rpm -Uvh /tmp/ec2-ami-tools.noarch.rpm && \
echo " + Updated ec2-ami-tools"

## Cleanup FileSystem
rm -f /tmp/ec2-ami-tools.noarch.rpm
rm -f /tmp/ec2-ami-tools.noarch.rpm.*

AMI

chmod +x /etc/cron.daily/do_amitools_update.sh

cat < <'YUM'> /etc/yum.repos.d/CentOS-Base.repo

# CentOS-Base.repo
#
# This file uses a new mirrorlist system developed by Lance Davis for CentOS.
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client. You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
baseurl=http://mirror.rightscale.com/centos/$releasever/os/$basearch/
http://mirrors.kernel.org/centos/$releasever/os/$basearch/
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
failovermethod=priority
gpgcheck=1
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[updates]
name=CentOS-$releasever - Updates
baseurl=http://mirror.rightscale.com/centos/$releasever/updates/$basearch/
http://mirrors.kernel.org/centos/$releasever/updates/$basearch/
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#packages used/produced in the build but not released
[addons]
name=CentOS-$releasever - Addons
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=addons
#baseurl=http://mirror.centos.org/centos/$releasever/addons/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

YUM

cat < <'EOF'> /root/.bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

EOF

cat < <'EOF'> /root/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
unset USERNAME

EOF

cat < <'EOF'> /root/.bash_logout
# ~/.bash_logout

clear

EOF

touch /root/.bash_logout

exit

EOL

chmod +x /mnt/ec2-fs/tmp/updates/install-script
chroot /mnt/ec2-fs/ /tmp/updates/install-script
echo "all Done"
echo "Cleaning up Image"
rm -fr /mnt/ec2-fs/tmp/updates
echo "Finished Step 4"
;;
"5")
echo "Prepping for upload"
sync
#umount -dlf /mnt/ec2-fs/proc
#umount -dlf /mnt/ec2-fs
echo "Bundling Image"
#ec2-bundle-image -i /mnt/image/$IMAGE_NAME -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_NUMBER
mkdir -p /mnt/tmp
ec2-bundle-vol -v /mnt/ec2-fs -d /mnt/tmp -p $IMAGE_NAME -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_NUMBER
echo "Finished Step 5"
;;
"6")
echo "Uploading"
ec2-upload-bundle -b $AWS_BUCKET -m /mnt/tmp/$IMAGE_NAME.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY
echo "Finished Step 6"
;;
"7")
echo "Cleanup"
umount /mnt/ec2-fs/proc
umount /mnt/ec2-fs
rm -fr /mnt/image/
rm -fr /mnt/ec2-fs
echo "File System Cleaned"
echo "Finished Step 7"
;;
"8")
exit
;;
esac
done

PERL ONE LINERS

2008-05-22T13:28:00.003+05:30

Level: Introductory

This article, as regular readers may have guessed, is the sequel to "One-liners 101," which appeared in a previous installment of "Cultured Perl". The earlier article is an absolute requirement for understanding the material here, so please take a look at it before you continue.

The goal of this article, as with its predecessor, is to show legible and reusable code, not necessarily the shortest or most efficient version of a program. With that in mind, let's get to the code!

Awk is commonly used for basic tasks such as breaking up text into fields; Perl excels at text manipulation by design. Thus, we come to our first one-liner, intended to add two columns in the text input to the script.

Listing 1. Like awk?

# add first and penultimate columns
# NOTE the equivalent awk script:
# awk '{i = NF - 1; print $1 + $i}'
perl -lane 'print $F[0] + $F[-2]'

So what does it do? The magic is in the switches. The -n and -a switches make the script a wrapper around input that splits the input on whitespace into the @F array; the -e switch adds an extra statement into the wrapper. The code of interest actually produced is:

Listing 2: The full Monty

while (<>)
{
@F = split(' ');
print $F[0] + $F[-2]; # offset -2 means "2nd to last element of the array"
}

Another common task is to print the contents of a file between two markers or between two line numbers.

Listing 3: Printing a range of lines

# 1. just lines 15 to 17
perl -ne 'print if 15 .. 17'

# 2. just lines NOT between line 10 and 20
perl -ne 'print unless 10 .. 20'

# 3. lines between START and END
perl -ne 'print if /^START$/ .. /^END$/'

# 4. lines NOT between START and END
perl -ne 'print unless /^START$/ .. /^END$/'

A problem with the first one-liner in Listing 3 is that it will go through the whole file, even if the necessary range has already been covered. The third one-liner does not have that problem, because it will print all the lines between the START and END markers. If there are eight sets of START/END markers, the third one-liner will print the lines inside all eight sets.

Preventing the inefficiency of the first one-liner is easy: just use the $. variable, which tells you the current line. Start printing if $. is over 15 and exit if $. is greater than 17.

Listing 4: Printing a numeric range of lines more efficiently

# just lines 15 to 17, efficiently
perl -ne 'print if $. >= 15; exit if $. >= 17;'

Enough printing, let's do some editing. Needless to say, if you are experimenting with one-liners, especially ones intended to modify data, you should keep backups. You wouldn't be the first programmer to think a minor modification couldn't possibly make a difference to a one-liner program; just don't make that assumption while editing the Sendmail configuration or your mailbox.

Listing 5: In-place editing

# 1. in-place edit of *.c files changing all foo to bar
perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

# 2. delete first 10 lines
perl -i.old -ne 'print unless 1 .. 10' foo.txt

# 3. change all the isolated oldvar occurrences to newvar
perl -i.old -pe 's{\boldvar\b}{newvar}g' *.[chy]

# 4. increment all numbers found in these files
perl -i.tiny -pe 's/(\d+)/ 1 + $1 /ge' file1 file2 ....

# 5. delete all but lines between START and END
perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

# 6. binary edit (careful!)
perl -i.bak -pe 's/Mozilla/Slopoke/g' /usr/local/bin/netscape

Why does 1 .. 10 specify line numbers 1 through 10? Read the "perldoc perlop" manual page. Basically, the .. operator iterates through a range. Thus, the script does not count 10 lines, it counts 10 iterations of the loop generated by the -n switch (see "perldoc perlrun" and Listing 2 for an example of that loop).

The magic of the -i switch is that it replaces each file in @ARGV with the version produced by the script's output on that file. Thus, the -i switch makes Perl into an editing text filter. Do not forget to use the backup option to the -i switch. Following the i with an extension will make a backup of the edited file using that extension.

Note how the -p and -n switch are used. The -n switch is used when you want explicitly to print out data. The -p switch implicitly inserts a print $_ statement in the loop produced by the -n switch. Thus, the -p switch is better for full processing of a file, while the -n switch is better for selective file processing, where only specific data needs to be printed.

Examples of in-place editing can also be found in the "One-liners 101" article.

Reversing the contents of a file is not a common task, but the following one-liners show than the -n and -p switches are not always the best choice when processing an entire file.

Listing 6: Reversal of files' fortunes

# 1. command-line that reverses the whole input by lines
# (printing each line in reverse order)
perl -e 'print reverse <>' file1 file2 file3 ....

# 2. command-line that shows each line with its characters backwards
perl -nle 'print scalar reverse $_' file1 file2 file3 ....

# 3. find palindromes in the /usr/dict/words dictionary file
perl -lne '$_ = lc $_; print if $_ eq reverse' /usr/dict/words

# 4. command-line that reverses all the bytes in a file
perl -0777e 'print scalar reverse <>' f1 f2 f3 ...

# 5. command-line that reverses each paragraph in the file but prints
# them in order
perl -00 -e 'print reverse <>' file1 file2 file3 ....

The -0 (zero) flag is very useful if you want to read a full paragraph or a full file into a single string. (It also works with any character number, so you can use a special character as a marker.) Be careful when reading a full file in one command (-0777), because a large file will use up all your memory. If you need to read the contents of a file backwards (for instance, to analyze a log in reverse order), use the CPAN module File::ReadBackwards. Also see "One-liners 101," which shows an example of log analysis with File::ReadBackwards.

Note the similarity between the first and second scripts in Listing 6. The first one, however, is completely different from the second one. The difference lies in using <> in scalar context (as -n does in the second script) or list context (as the first script does).

The third script, the palindrome detector, did not originally have the $_ = lc $_; segment. I added that to catch those palindromes like "Bob" that are not the same backwards.

My addition can be written as $_ = lc; as well, but explicitly stating the subject of the lc() function makes the one-liner more legible, in my opinion.

Listing 7: Rewrite with a random number

# replace string XYZ with a random number less than 611 in these files
perl -i.bak -pe "s/XYZ/int rand(611)/e" f1 f2 f3

This is a filter that replaces XYZ with a random number less than 611 (that number is arbitrarily chosen). Remember the rand() function returns a random number between 0 and its argument.

Note that XYZ will be replaced by a different random number every time, because the substitution evaluates "int rand(611)" every time.

Listing 8: Revealing the files' base nature

# 1. Run basename on contents of file
perl -pe "s@.*/@@gio" INDEX

# 2. Run dirname on contents of file
perl -pe 's@^(.*/)[^/]+@$1\n@' INDEX

# 3. Run basename on contents of file
perl -MFile::Basename -ne 'print basename $_' INDEX

# 4. Run dirname on contents of file
perl -MFile::Basename -ne 'print dirname $_' INDEX

One-liners 1 and 2 came from Paul, while 3 and 4 were my rewrites of them with the File::Basename module. Their purpose is simple, but any system administrator will find these one-liners useful.

Listing 9: Moving or renaming, it's all the same in UNIX

# 1. write command to mv dirs XYZ_asd to Asd
# (you may have to preface each '!' with a '\' depending on your shell)
ls | perl -pe 's!([^_]+)_(.)(.*)!mv $1_$2$3 \u$2\E$3!gio'

# 2. Write a shell script to move input from xyz to Xyz
ls | perl -ne 'chop; printf "mv $_ %s\n", ucfirst $_;'

For regular users or system administrators, renaming files based on a pattern is a very common task. The scripts above will do two kinds of job: either remove the file name portion up to the _ character, or change each filename so that its first letter is uppercased according to the Perl ucfirst() function.

There is a UNIX utility called "mmv" by Vladimir Lanin that may also be of interest. It allows you to rename files based on simple patterns, and it's surprisingly powerful. See the Resources section for a link to this utility.

Some of mine

The following is not a one-liner, but it's a pretty useful script that started as a one-liner. It is similar to Listing 7 in that it replaces a fixed string, but the trick is that the replacement itself for the fixed string becomes the fixed string the next time.

The idea came from a newsgroup posting a long time ago, but I haven't been able to find original version. The script is useful in case you need to replace one IP address with another in all your system files -- for instance, if your default router has changed. The script includes $0 (in UNIX, usually the name of the script) in the list of files to rewrite.

As a one-liner it ultimately proved too complex, and the messages regarding what is about to be executed are necessary when system files are going to be modified.

Listing 10: Replace one IP address with another one

#!/usr/bin/perl -w

use Regexp::Common qw/net/; # provides the regular expressions for IP matching

my $replacement = shift @ARGV; # get the new IP address

die "You must provide $0 with a replacement string for the IP 111.111.111.111"
unless $replacement;

# we require that $replacement be JUST a valid IP address
die "Invalid IP address provided: [$replacement]"
unless $replacement =~ m/^$RE{net}{IPv4}$/;

# replace the string in each file
foreach my $file ($0, qw[/etc/hosts /etc/defaultrouter /etc/ethers], @ARGV)
{
# note that we know $replacement is a valid IP address, so this is
# not a dangerous invocation
my $command = "perl -p -i.bak -e 's/111.111.111.111/$replacement/g' $file";

print "Executing [$command]\n";
system($command);
}

Note the use of the Regexp::Common module, an indispensable resource for any Perl programmer today. Without Regexp::Common, you will be wasting a lot of time trying to match a number or other common patterns manually, and you're likely to get it wrong.

Handy One Liners for SED COmmand

2008-05-22T12:40:00.000+05:30

Handy one-liners for SED

HANDY ONE-LINERS FOR SED (Unix stream editor)

Latest version of this file is usually at:
http://www.student.northpark.edu/pemente/sed/sed1line.txt
http://www.cornerstonemag.com/sed/sed1line.txt

FILE SPACING:

# double space a file
sed G

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed '/^$/d;G'

# triple space a file
sed 'G;G'

# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'

NUMBERING:

# number each line of a file (simple left alignment). Using a tab (see
# note on '\t' at end of file) instead of space will preserve margins.
sed = filename | sed 'N;s/\n/\t/'

# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *$.\{6,\}$\n/\1 /'

# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'

# count lines (emulates "wc -l")
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$//" # method 1
sed -n p # method 2

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of sed. Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file

# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file

# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'

# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'

# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space

# center all text in the middle of 79-column width. In method 1,
# spaces at the beginning of the line are significant, and trailing
# spaces are appended at the end of the line. In method 2, spaces at
# the beginning of the line are discarded in centering the line, and
# no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/$ *$\1/\1/' # method 2

# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/$.*$foo$.*foo$/\1bar\2/' # replace the next-to-last case
sed 's/$.*$foo/\1bar/' # replace only the last case

# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'

# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only

# reverse order of lines (emulates "tac")
# bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2

# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/$.$$.*\n$/&\2\1/;//D;s/.//'

# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'

# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'

# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/$.*[0-9]$$[0-9]\{3\}$/\1,\2/;ta' # other seds

# add commas to numbers with decimal points and minus signs (GNU sed)
gsed ':a;s/$^\|[^0-9.]$$[0-9]\+$$[0-9]\{3\}$/\1\2,\3/g;ta'

# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds

SELECTIVE PRINTING OF CERTAIN LINES:

# print first 10 lines of file (emulates behavior of "head")
sed 10q

# print first line of file (emulates "head -1")
sed q

# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'

# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'

# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2

# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2

# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax

# print the line immediately before a regexp, but not the line
# containing the regexp
sed -n '/regexp/{g;1!p;};h'

# print the line immediately after a regexp, but not the line
# containing the regexp
sed -n '/regexp/{n;p;}'

# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'

# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'

# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed only

# print paragraph if it contains AAA (blank lines separate paragraphs)
# HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only

# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'

# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax

# print section of file from regular expression to end of file
sed -n '/regexp/,$p'

# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2

# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files

# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds

# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'

# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^$.*$\n\1$/!P; D'

# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^$[ -~]*\n$.*\n\1/d; s/\n//; h; P'

# delete the first 10 lines of a file
sed '1,10d'

# delete the last line of a file
sed '$d'

# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'

# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2

# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds

# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2

# delete all CONSECUTIVE blank lines from file except the first; also
# deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF

# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'

# delete all leading blank lines at top of file
sed '/./,$!d'

# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02*

# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:

# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed v1.5

# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line

# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line

# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'

# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'

# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'

# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2

# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^$.*$\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:

cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk

For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult "sed &
awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,
1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty
and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst
distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power
of sed, one must understand "regular expressions." For this, see
"Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997).
The manual ("man") pages on Unix systems may be helpful (try "man
sed", "man regexp", or the subsection on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach sed use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.

QUOTING SYNTAX: The preceding examples use single quotes ('...')
instead of double quotes ("...") to enclose editing commands, since
sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes
(`...`), which are expanded by the shell if they are enclosed in
double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., \!) to
properly run the examples listed above, even within single quotes.
Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.

USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
However, most versions of sed do not recognize the '\t' abbreviation,
so when typing these scripts from the command line, you should press
the TAB key instead. '\t' is supported as a regular expression
metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.

VERSIONS OF SED: Versions of sed do differ, and some slight syntax
variation is to be expected. In particular, most do not support the
use of labels (:name) or branch instructions (b,t) within editing
commands, except at the end of those commands. We have used the syntax
which will be portable to most users of sed, even though the popular
GNU versions of sed allow a more succinct syntax. When the reader sees
a fairly long command such as this:

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d

it is heartening to know that GNU sed will let you reduce it to:

sed '/AAA/b;/BBB/b;/CCC/b;d' # or even
sed '/AAA\|BBB\|CCC/b;d'

In addition, remember that while many versions of sed accept a command
like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which
contains space before the 's'. Omit the space when typing the command.

OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the "find" expression is specified before
giving the "s/.../.../" instruction. Thus:

sed 's/foo/bar/g' filename # standard replace command
sed '/foo/ s/foo/bar/g' filename # executes more quickly
sed '/foo/ s//bar/g' filename # shorthand sed syntax

On line selection or deletion in which you only need to output lines
from the first part of the file, a "quit" command (q) in the script
will drastically reduce processing time for large files. Thus:

sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster

Anand Shah
Rediff.com

How does ssl work

2008-03-26T22:16:00.002+05:30

Internet communication typically runs through multiple program layers on a server before getting to the requested data such as a web page or cgi scripts.

The outer layer is the first to be hit by the request. This is the high level protocols such as HTTP (web server), IMAP (mail server), and FTP (file transfer).

Determining which outer layer protocol will handle the request depends on the type of request made by the client. This high level protocol then processes the request through the Secure Sockets Layer. If the request is for a non-secure connection it passes through to the TCP/IP layer and the server application or data.

If the client requested a secure connection the ssl layer initiates a handshake to begin the secure communication process. Depending on the SSL setup on the server, it may require that a secure connection be made before allowing communication to pass through to the TCP/IP layer in which case a non-secure request will send back an error asking for them to retry securely (or simply deny the non-secure connection).

This is a good time to answer a question we periodically get: "what does ssl encrypt?"

This question is usually geared toward whether or not the path and query string is encrypted in an HTTPS "get" request (this is where form field responses or program variables are tagged on to the end of the url). These fields are stripped off of the URL when creating the routing information in the https packaging process by the browser and are included in the encrypted data block.

The page data (form, text, and query string) are passed in the encrypted block after the encryption methods are determined and the handshake completes.

A related issue that frequently comes up is whether or not form data is transmited with encryption if the blank form is displayed without https. If the form "action" is set to use https then the ssl handshake will take place before the data is sent. Whether or not the original form is displayed using https has little to do with the form submission unless the form action uses a relative path, in which case the default will be to use the protocol that was used to display the form.

PHP SESSION HANDLING EXAMPLE

2008-03-26T22:11:00.000+05:30

The below code is a very simple example of Session.
Please note that the "session_start" statement *must* be the first line of your code.
To see the example at work, please review:

http://www.xn--ovg.com/session

tedd

--- copy below and save as session1.php -----
session_start();
?>

"http://www.w3.org/TR/html4/strict.dtd">

Stuff by tedd

Anand's session stuff (page 1)

PHPSESSID =

Click the next page and see numbr of visits during this visit.

Next page

--- copy below and save as session2.php -----

session_start();
($_SESSION['count']) ? $_SESSION['count']++ : $_SESSION['count'] = 1;
?>

"http://www.w3.org/TR/html4/strict.dtd">

Stuff by tedd

Anand's session stuff (page 2)

PHPSESSID =

You have been here times in this session.

Test the SMTP Service on WIndows

2008-03-26T22:03:00.000+05:30

Test the SMTP Service

To test the SMTP service, follow these steps:
1. On a computer running Windows Server 2003, type Telnet at a command prompt, and then press ENTER.
2. At the telnet prompt, type set LocalEcho, press ENTER, and then type open 25, and then press ENTER.

The output resembles the following:

220 computername.microsoft.com ESMTP Server (Microsoft Exchange Internet Mail Service 5.5.2651.58) ready

3. Type helo me, and then press ENTER.

The output resembles the following:

250 OK

4. Type mail from:email@domain.com, and then press ENTER.

The output resembles the following:

250 OK - mail from

5. Type rcpt to:youremail@yourdomain.com, and then press ENTER.

The output resembles the following:

250 OK - Recipient

6. Type Data, and then press ENTER.

The output resembles the following:

354 Send data. End with CRLF.CRLF

7. Type Subject:This is a test, and then press ENTER two times.
8. Type Testing, and then press ENTER.
9. Press ENTER, type a period (.), and then press ENTER.

The output resembles the following:

250 OK

10. Type quit, and then press ENTER.
The output resembles the following:

221 Closing Port / Mail queued for delivery

Disallow download .MDB file

2008-03-26T22:02:00.000+05:30

Disallow download of certain file type

If what you want to do is prevent a download of your .mdb file,
just create an App_Data directory and place the Access database in it.

Files stored in the App_Data folder are not returned in response to direct HTTP requests,
which makes the App_Data folder the recommended location for data stored with your
application, including .mdf (SQL Server Express Edition), .mdb (Microsoft Access), or XML files.

Command Line Examples linux

2008-03-26T21:58:00.001+05:30

Linux command line reference for common operations.

This is a linux command line reference for common operations.
Examples marked with • are valid/safe to paste without modification into a terminal, so
you may want to keep a terminal window open while reading this so you can cut & paste.
All these commands have been tested both on Fedora and Ubuntu.

Command Description
• apropos whatis Show commands pertinent to string. See also threadsafe
• man -t man | ps2pdf - > man.pdf make a pdf of a manual page
which command Show full path name of command
time command See how long a command takes
• time cat Start stopwatch. Ctrl-d to stop. See also sw
• nice info Run a low priority command (The "info" reader in this case)
• renice 19 -p $$ Make shell (script) low priority. Use for non interactive tasks
dir navigation
• cd - Go to previous directory
• cd Go to $HOME directory
(cd dir && command) Go to dir, execute command and return to current dir
• pushd . Put current dir on stack so you can popd back to it
file searching
• alias l='ls -l --color=auto' quick dir listing
• ls -lrt List files by date. See also newest and find_mm_yyyy
• ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
find -name '*.[ch]' | xargs grep -E 'expr' Search 'expr' in this dir and below. See also findrepo
find -type f -print0 | xargs -r0 grep -F 'example' Search all regular files for 'example' in this dir and below
find -maxdepth 1 -type f | xargs grep -F 'example' Search all regular files for 'example' in this dir
find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
• find -type f ! -perm -444 Find files not readable by all (useful for web site)
• find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
• locate -r 'file[^/]*\.txt' Search cached index for names. This re is like glob *file*.txt
• look reference Quickly search (sorted) dictionary for prefix
• grep --color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
gpg -c file Encrypt file
gpg file.gpg Decrypt file
tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
tar -c dir/ | gzip | gpg -c | ssh user@remote 'dd of=dir.tar.gz.gpg' Make encrypted archive of dir/ on remote machine
find dir/ -name '*.txt' | tar -c --files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
find dir/ -name '*.txt' | xargs cp -a --target-directory=dir_txt/ --parents Make copy of subset of dir/ and below
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
( tar -c /dir/to/copy ) | ssh -C user@remote 'cd /where/to/ && tar -x -p' Copy (with permissions) copy/ dir to remote:/where/to/ dir
dd bs=1M if=/dev/sda | gzip | ssh user@remote 'dd of=sda.gz' Backup harddisk to remote machine
rsync (Use the --dry-run option for testing)
rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
rsync --bwlimit=1000 fromfile tofile Locally copy with rate limit. It's like nice for I/O
rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html' Mirror web site (using compression and encryption)
rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
• ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER's home directory on $HOST
ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
wget (multi purpose download tool)
• (cd cli && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/ Download a set of files to the current directory
wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
• wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head Process output directly
echo 'wget url' | at 01:00 Download url at 1AM to current dir
wget --limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
wget -nv --spider --force-html -i bookmarks.html Check links in a file
wget --mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
ethtool eth0 Show status of ethernet interface eth0
ethtool --change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
iwconfig eth1 Show status of wireless interface eth1
iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
• iwlist scan List wireless networks in range
• ip link show List network interfaces
ip link set dev eth0 name wan Rename interface eth0 to wan
ip link set dev eth0 up Bring interface eth0 up (or down)
• ip addr show List addresses for interfaces
ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
• ip route show List routing table
ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
• tc qdisc add dev lo root handle 1:0 netem delay 20msec Add 20ms latency to loopback device (for testing)
• tc qdisc del dev lo root Remove latency added above
• host pixelbeat.org Lookup DNS ip address for name or vice versa
• hostname -i Lookup local ip address (equivalent to host `hostname`)
• whois pixelbeat.org Lookup whois info for hostname or ip address
• netstat -tupl List internet services on a system
• netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
• smbtree Find windows machines. See also findsmb
nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
smbclient -L windows_box List shares on windows machine or samba server
mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
echo 'message' | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout, so if you want to edit files, append newfile)
sed 's/string1/string2/g' Replace string1 with string2
sed 's/$.*$1/\12/g' Modify anystring1 to anystring2
sed '/ *#/d; /^ *$/d' Remove comments and blank lines
sed ':a; /\\$/N; s/\\\n//; ta' Concatenate lines with trailing \
sed 's/[ \t]*$//' Remove trailing spaces from lines
sed 's/$[\\`\\"$\\\\]$/\\\1/g' Escape shell metacharacters active within double quotes
• seq 10 | sed "s/^/ /; s/ *$.\{7,\}$/\1/" Right align numbers
sed -n '1000p;1000q' Print 1000th line
sed -n '10,20p;20q' Print lines 10 to 20
sed -n 's/.*(title)$.*$<\/title>.*/\1/ip;T;q' Extract title from HTML web page
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses
• echo 'Test' | tr '[:lower:]' '[:upper:]' Case conversion
• tr -dc '[:print:]' < /dev/urandom Filter non printable characters
• history | wc -l Count lines
set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)
sort file1 file2 | uniq Union of unsorted files
sort file1 file2 | uniq -d Intersection of unsorted files
sort file1 file1 file2 | uniq -u Difference of unsorted files
sort file1 file2 | uniq -u Symmetric Difference of unsorted files
join -a1 -a2 file1 file2 Union of sorted files
join file1 file2 Intersection of sorted files
join -v2 file1 file2 Difference of sorted files
join -v1 -v2 file1 file2 Symmetric Difference of sorted files
math
• echo '(1 + sqrt(5))/2' | bc -l Quick math (Calculate φ). See also bc
• echo 'pad=20; min=64; (100*10^6)/((pad+min)*8)' | bc More complex (int) e.g. This shows max FastE packet rate
• echo 'pad=20; min=64; print (100E6)/((pad+min)*8)' | python Python handles scientific notation
• echo 'pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)' | gnuplot -persist Plot FastE packet rate vs packet size
• echo 'obase=16; ibase=10; 64206' | bc Base conversion (decimal to hexadecimal)
• echo $((0x2dec)) Base conversion (hex to dec) ((shell arithmetic expansion))
• units -t '100m/9.74s' 'miles/hour' Unit conversion (metric to imperial)
• units -t '500GB' 'GiB' Unit conversion (SI to IEC prefixes)
• units -t '1 googol' Definition lookup
• seq 100 | (tr '\n' +; echo 0) | bc Add a column of numbers. See also add and funcpy
calendar
• cal -3 Display a calendar
• cal 9 1752 Display a calendar for a particular month year
• date -d fri What date is it this friday. See also day
• date --date='25 Dec' +%A What day does xmas fall on, this year
• date --date '1970-01-01 UTC 2147483647 seconds' Convert number of seconds since the epoch to a date
• TZ=':America/Los_Angeles' date What time is it on West coast of US (use tzselect to find TZ)
echo "mail -s 'get the train' P@draigBrady.com < /dev/null" | at 17:45 Email reminder
• echo "DISPLAY=$DISPLAY xmessage cooker" | at "NOW + 30 minutes" Popup reminder
locales
• printf "%'d\n" 1234 Print number with thousands grouping appropriate to locale
• BLOCK_SIZE=\'1 ls -l get ls to do thousands grouping appropriate to locale
• echo "I live in `locale territory`" Extract info from locale database
• LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes
• locale | cut -d= -f1 | xargs locale -kc | less List fields available in locale database
recode (Obsoletes iconv, dos2unix, unix2dos)
• recode -l | less Show available conversions (aliases on each line)
recode windows-1252.. file_to_change.txt Windows "ansi" to local charset (auto does CRLF conversion)
recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset
recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8
recode ../b64 < file.txt > file.b64 Base64 encode
recode /qp.. < file.txt > file.qp Quoted printable decode
recode ..HTML < file.txt > file.html Text to HTML
• recode -lf windows-1252 | grep euro Lookup table of characters
• echo -n 0x80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap
• echo -n 0x20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding
• echo -n 0x20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding
CDs
gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom
mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir
mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only)
cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW
gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom - Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
cdparanoia -B Rip audio tracks from CD to wav files in current dir
cdrecord -v dev=/dev/cdrom -audio *.wav Make audio CD from all wavs in current dir (see also cdrdao)
oggenc --tracknum='track' track.cdda.wav -o 'track.ogg' Make ogg file from wav file
disk space (See also FSlint)
• ls -lSr Show files by size, biggest last
• du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop
• df -h Show free space on mounted filesystems
• df -i Show free inodes on mounted filesystems
• fdisk -l Show disks partitions sizes and types (run as root)
• rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n List all packages by installed size (Bytes) on rpm distros
• dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n List all packages by installed size (KBytes) on deb distros
• dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate
monitoring/debugging
• tail -f /var/log/messages Monitor messages in a log file
• strace -c ls >/dev/null Summarise/profile system calls made by command
• strace -f -e open ls >/dev/null List system calls made by command
• ltrace -f -e getenv ls >/dev/null List library calls made by command
• lsof -p $$ List paths that process id has open
• lsof ~ List processes that have specified path open
• tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me
• ps -e -o pid,args --forest List processes in a hierarchy
• ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' List processes by % cpu usage
• ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem usage. See also ps_mem.py
• ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process
• ps -p 1,2 List info for particular process IDs
• last reboot Show system reboot history
• free -m Show amount of (remaining) RAM (-m displays in MB)
• watch -n1 'cat /proc/interrupts' Watch changeable data continuously
system information (see also sysinfo) ('#' means root access is required)
• uname -a Show kernel version and system architecture
• head -n1 /etc/issue Show name and version of distribution
• cat /proc/partitions Show all partitions registered on the system
• grep MemTotal /proc/meminfo Show RAM total seen by the system
• grep "model name" /proc/cpuinfo Show CPU(s) info
• lspci -tv Show PCI info
• lsusb -tv Show USB info
• mount | column -t List mounted filesystems on the system (and align output)
# dmidecode -q | less Display SMBIOS/DMI information
# smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total
# hdparm -i /dev/sda Show info about disk sda
# hdparm -tT /dev/sda Do a read speed test on disk sda
# badblocks -s /dev/sda Test for unreadable blocks on disk sda
interactive (see also linux keyboard shortcuts)
• readline Line editor used by bash, python, bc, gnuplot, ...
• screen Virtual terminals with detach capability, ...
• mc Powerful file manager that can browse rpm, tar, ftp, ssh, ...
• gnuplot Interactive/scriptable graphing
• links Web browser
miscellaneous
• alias hd='od -Ax -tx1z -v' Handy hexdump. (usage e.g.: • hd /proc/self/cmdline | less)
• alias realpath='readlink -f' Canonicalize path. (usage e.g.: • realpath ~/../$USER)
• set | grep $USER Search current environment
touch -c -t 0304050607 file Set file timestamp (YYMMDDhhmm)

####################################################################################

World of FIND Commands in LINUX

2008-03-26T21:56:00.000+05:30

FIND COMMANDS

sudo find / -type f -name *.jpg -exec cp {} . \;

find . -type f -size +10000 -exec ls -al {} \;
find . -atime +1 -type f -exec mv {} TMP \; # mv files older then 1 day to dir TMP
find . -name "-F" -exec rm {} \; # a script error created a file called -F
find . -exec grep -i "vds admin" {} \;
find . \! -name "*.Z" -exec compress -f {} \;
find . -type f \! -name "*.Z" \! -name ".comment" -print | tee -a /tmp/list
find . -name *.ini
find . -exec chmod 775 {} \;
find . -user xuser1 -exec chown -R user2 {} \;
find . -name ebtcom*
find . -name mkbook
find . -exec grep PW0 {} \;
find . -exec grep -i "pw0" {} \;
find . -atime +6
find . -atime +6 -exec ll | more
find . -atime +6 -exec ll | more \;
find . -atime +6 -exec ll \;
find . -atime +6 -exec ls \;
find . -atime +30 -exec ls \;
find . -atime +30 -exec ls \; | wc -l
find . -name auth*
find . -exec grep -i plotme10 {};
find . -exec grep -i plotme10 {} \;
find . -ls -exec grep 'PLOT_FORMAT 22' {} \;
find . -print -exec grep 'PLOT_FORMAT 22' {} \;
find . -print -exec grep 'PLOT_FORMAT' {} \;
find . -print -exec grep 'PLOT_FORMAT' {} \;
find ./machbook -exec chown 184 {} \;
find . \! -name '*.Z' -exec compress {} \;
find . \! -name "*.Z" -exec compress -f {} \;
find /raid/03c/ecn -xdev -type f -print
find /raid/03c/ecn -xdev -path -type f -print
find / -name .ssh* -print | tee -a ssh-stuff
find . -name "*font*"
find . -name hpmcad*
find . -name *fnt*
find . -name hp_mcad* -print
find . -grep Pld {} \;
find . -exec grep Pld {} \;
find . -exec grep Pld {} \;
find . -exec grep PENWIDTH {} \; | more
find . -name config.pro
find . -name config.pro
find /raid -type d ".local_sd_customize" -print
find /raid -type d -name ".local_sd_customize" -print
find /raid -type d -name ".local_sd_customize" -ok cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;
find /raid -type d -name ".local_sd_customize" -exec cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;
find . -name xeroxrelease
find . -exec grep xeroxrelease {} \;
find . -name xeroxrelease
find . -name xeroxrelease* -print 2>/dev/null
find . -name "*release*" 2>/dev/null
find / -name "*xerox*" 2>/dev/null
find . -exec grep -i xeroxrelease {} \;
find . -print -exec grep -i xeroxrelease {} \;
find . -print -exec grep -i xeroxrelease {} \; > xeroxrel.lis
find . -exec grep -i xeroxrel {} \;
find . -print -exec grep -i xeroxrel {} \;
find . -print -exec grep -i xeroxrel {} \; | more
find /raid/03c/inwork -xdev -type f -print >> /raid/04d/user_scripts/prt_list.tmp
find . -exec grep '31.53' {} \;
find . -ls -exec grep "31/.53" {} \; > this.lis
find . -print -exec grep "31/.53" {} \; > this.lis
find . -print -exec grep 31.53 {} \; > this.lis
find . -exec grep -i pen {} /;
find . -exec grep -i pen {} \;
find . -print -exec grep -i pen {} \; | more
find . -exec grep -i pen {} \;
find . -atime +6 -exec ll | more \;
find . -atime +6 -exec ll \;
find . -atime +6 -exec ls \;
find . -atime +30 -exec ls \;
find . -atime +30 -exec ls \; | wc -l
find . \! -name '*.Z' -exec compress -f {} \;
find . -name 'cache*' -depth -exec rm {} \;
find . -name 'cache*' -depth -print | tee -a /tmp/cachefiles
find . -name 'cache[0-9][0-9]*' -depth -print | tee -a /tmp/cachefiles
find . -name 'hp_catfile' 'hp_catlock' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_catfile' -name 'hp_catlock' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_cat*' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_cat[fl]*' -depth -print | tee -a /tmp/hp.cats
find /raid -name 'hp_cat[fl]*' -depth -print
find . \! -name '*.Z' -exec compress -f {} \;
find . -name '*' -exec compress -f {} \;
find . -xdev -name "wshp1*" -print
find . -xdev -name "wagoneer*" -print
find . -name "xcmd" -depth -print
find /usr/contrib/src -name "xcmd" -depth -print
find /raid -type d -name ".local_sd_customize" -exec ls {} \;
find /raid -type d -name ".local_sd_customize" \
-exec cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;

Qmail Tip sand Tricks 2

2008-03-26T21:47:00.004+05:30

Mail to Valid Users Is Bouncing or Disappearing

If you use users/assign as described in Chapter 15, a common mistake is to add a user to the system without updating the users file. Fortunately, this oversight is easily remedied:

# cd /var/qmail/users; make

Delivering Mail on Intermittent Connections

If your qmail system is a hub host for remote systems that connect intermittently by dialup, it is straightforward but messy to deliver the mail while the remote systems are connected.

One approach is to create a flag file in a known directory when a host connects and delete the file when the host disconnects. Then run a script periodically from cron that loops over all of the flag files to push out mail to currently connected hosts.

To flesh out this example, assume there are three dialup hosts called red.example.com, blue.example.com, and green.example.com. Create virtualdomains that give them different virtual domain prefixes:

red.example.com:alias-dial-red
blue.example.com:alias-dial-blue
green.example.com:alias-dial-green

You can put all of the alias-dial mail into one Maildir since the Delivered-To: prefixes keep them separate. To put all the mail for the three hosts into ~alias/dialmail/, create ~alias/.qmail-dial-default containing the line ./dialmail/.

To track the currently connected hosts, put the flag files into ~alias/dialflags and have the dialup connection script create a file with the host's simple name (red, blue, or green) in that directory containing the host's current IP address. Then run this script from cron to push out the mail to whichever hosts are currently connected:

#!/bin/sh
# run this every 15 minutes from cron to push out the mail

cd /var/qmail/alias/dialflags

for hn in *
do
ip=$(cat $hn) # IP address in the flag file

setlock ../$hn.lock \ # lock deliveries to this host
maildirsmtp /var/qmail/alias/dialmail \
alias-dial-$hn- $ip my.example.com 2>&1 |
splogger serial
done

If you also want to push out any waiting mail as soon as a host connects, also put a call to maildirsmtp into the host's connection script. Be sure to use the same lock file to avoid confusion if the cron job happens to run at the same time. If you add another host called purple, you only need to add another line to virtualdomains:

purple.example.com:alias-dial-purple

The remote hosts can use a similar setup to forward their mail to the main host, using a single smarthost entry in virtualdomains.

Code in extra.h to copy everything to log

#define QUEUE_EXTRA "Tlog\0"
#define QUEUE_EXTRALEN 5

Now every message will be copied to the address log, so you can create ~alias/.qmail-log to save the mail:

./logmaildir/

The .qmail file must save the mail but cannot forward it. Why not? Because forwarding mail invokes qmail-queue again, which will redeliver the mail to log, creating a nasty mail loop.

Deleting Stale Mail

cd /home
{
# unread mail over a month old
find /home/*/Maildir/new -type f -mtime +30 -p
# read mail over three months
find /home/*/Maildir/cur -type f -mtime +90 -p
# any mail marked deleted
find */Maildir -type f -name "*:2,*T*" -print
# any mail in Trash/new or cur
find */Maildir/.Trash/??? -type f -print
} | xargs -t rm

Web Sites

There are several excellent sources of qmail information online.

http://cr.yp.to

Dan Bernstein's web site, the official source for qmail and all of his ad-on packages.

http://www.qmail.org

Russ Nelson's qmail resource site, intended to have links to all of the other resources on the Web.

http://qmail.gurus.com

The author's companion site for this book, containing scripts, updates and corrections, links to other resources, and ordering info for more copies.

http://www.lifewithqmail.org

Dave Sill's Life with qmail, an online guide to setting up and using qmail. It offers specific advice about where to install qmail, and where to put all of the files and directories that qmail needs. This is by far the most widely used setup and the one that qmail experts are the most familiar with, so it's the one you should use. The file and directory locations used in this book are consistent with these.

http://www.lifewithqmail.org/ldap/

Henning Brauer's Life with qmail-ldap, a guide to setting up qmail-ldap. Indispensable for qmail-ldap users.

http://www.ezmlm.org

The home page for the ezmlm-idx mailing list manager, with software and documentation.

http://tinydns.org

Russ Nelson's site for Dan Bernstein's djbdns, a DNS package that relates to BIND roughly as qmail relates to sendmail. Not required for qmail, but if you're setting up a DNS server along with your mail server, it's probably the software you want to use.

Qmail Tips and Tricks

2008-03-26T21:42:00.003+05:30

Qmail Won't Compile

You have unpacked the qmail sources and typed make, but it won't compile. If you're receiving error messages about errno, you've run into a compatibility problem between qmail and recent versions of the GNU C library. The fix is very simple. See Building with Recent GLIBC and Fixing the errno Problem in Chapter 3.

(This is the number one question on the qmail mailing list, so frequent that there's an autoresponder that mails back the answer to any message that contains the word "errno".)

Why Qmail Is Delivering Mail Very Slowly

If qmail seems to wait about half a minute to do anything when you inject mail, the problem is almost certainly that the lock/trigger file used to communicate between qmail-queue and qmail-send is messed up. That file should be a named pipe:

# ls -l /var/qmail/queue/lock/trigger
prw--w--w- 1 qmails qmail 0 Nov 7 03:02 /var/qmail/queue/lock/trigger

If it's a regular file or anything other than a pipe, you have a problem. Fortunately, it's a problem that's easy to fix:

# svc -td /service/qmail-send # shut qmail down for a minute
# tail -f /service/qmail-send/log/main/current
# # wait until the log says that it's exited
# rm /var/qmail/queue/lock/trigger # remove bogus trigger
# cd wherever you built qmail from source
# make setup check # recreates all the crucial files including trigger
# svc -u /service/qmail-send # restart qmail

This is the second most frequently asked question on the qmail mailing list, and tends to get aggrieved responses pointing out that the answer is in the archives about a hundred times. So don't ask it, because now you know the answer.

Daemons Won't Start, or They Start and Crash Every Few Seconds

Starting a daemon under svscan and supervise is simple in concept, although the details can bite you. The super-daemon is started at system boot time by running /command/svscanboot. It runs svscan to control daemons and the useful but obscure readproctitle, which takes any error messages from svscan and puts them into its command area so that ps will show it.[1]

[1] This odd way of displaying error messages is intended to work even in the presence of serious configuration screwups like disks that should be mounted but aren't and directories that are supposed to be writable but aren't.

Every five seconds svscan looks at all of the subdirectories of /service and starts up a supervise process on any that don't have one running. In the usual case that the subdirectory in turn has a subdirectory called log, it starts a second supervise process in the subdirectory and pipes the output from the first process to the second.

When supervise starts up a daemon, it runs the file run in the daemon's directory. That file has to be a runnable program that either is or, more commonly, exec's the daemon itself. That means that run has to have its execute bits set and, if it's a shell script, start with #!/bin/sh so that it's runnable. If either of those isn't the case, there is a failed attempt to start the daemon every five seconds. A ps l that shows readproctitle should reveal the error messages and give hints about what needs to be fixed.

The run script generally sets up the program environment and then exec's the actual daemon. If you become super-user and type ./run, the daemon should start. If that works, the daemon still doesn't start, and you don't use full program paths in the run file, the problem is most likely that the search path that supervise uses isn't the same as the one you're using. Look at /command/svscanboot to see the search patch that it uses. Most notably, it does not include /var/qmail/bin unless you edit the file yourself to include it.

Nothing Gets Logged

Sometimes the daemon runs but nothing's going into the log files. This generally is due to either file protection problems or an incorrect set of multilog options. The usual way to run multilog is to create a subdirectory called main in which it rotates log files. It's safer to run daemons as a user other than root, so when possible, use qmaill, the qmail log user. A common error is to forget to change the ownership of the log file directory to qmaill (or whatever the log user is). When multilog starts successfully, it creates a current log file in the directory, so if there's no main/current, the most likely problem is directory ownership or protection.

If multilog is running but there's nothing logged, the most likely problems are that the daemon isn't sending anything to log, or that multilog's options are telling it to discard everything. Because the daemon and the logger are connected with a regular Unix pipe, only messages sent to the daemon's standard output go to the logger. In particular, anything sent to standard error shows up in readproctitle, not the log. If, as is usually the case, you want to log the errors a daemon reports, just redirect the error output to the standard output in the run script with the standard shell redirect 2>&1. (That redirect is at the end of just about every run script example in this book.)

If the daemon is a program originally intended to run as a standalone daemon rather than under daemontools, it probably sends its reports to syslog, not to standard output or standard error. In most cases, there is an option to send messages to stdout or stderr.

If you are using multilog options to select what to log, be sure that you're selecting what you think you are. In particular, its pattern language resembles shell wildcards but is in fact considerably weaker because it doesn't move ahead or back up on a failed match. (Patterns do resemble shell wildcards closely enough that they should always be quoted to keep the shell from messing with them.) The pattern must match the whole line, and stars stop matching the moment they see the following character in the pattern. If a pattern is, say, +'+*: status: *', it will match one: status: two, but it will not match one: two: status: three, because the star will stop at the first colon and won't look for the second one. If the pattern didn't have the star at the end, it wouldn't match anything useful because it wouldn't match any lines with anything after the status:. In practice, most log file messages have a pretty simple syntax, and it's not hard to come up with adequate patterns if you keep in mind the limitations of the pattern-matching language. For debugging, start with no patterns to be sure that the stream of messages going into the log files contains what you expect, then add one or two patterns at a time and restart multilog with svc -t and see what's going into main/current each time until it looks right.

Daemons Are Running but Making No Progress

One of the most baffling problems occurs when the daemon seems OK, the logger seems OK, but the daemon's not doing anything. What's wrong? Usually the problem is that the disk to which the log files are written has filled up or is mounted read-only. Because multilog is designed not to lose any log data, if it can't write to the disk, it just waits and retries until it can. This means that the pipe between the daemon and multilog fills up and the daemon stalls waiting to be able to write to the pipe. The solution is to delete some files and fix whatever it was that filled up the disk so it doesn't happen again. If the disk is full of files written by various multilog loggers, adding or adjusting s and n options to set the maximum size and number of log files can help.

Mail Rejected with Stray Newline Reports

The SMTP spec says that the way that each line of text in an SMTP session ends is with a carriage return/line feed pair (0d 0a in hex or \r\n in C.) Some buggy MUAs and MTAs only try to send mail that contains linefeeds with no preceding carriage return. Qmail's SMTP daemon normally rejects such mail with a log message like Stray newline from 10.2.3.4 because there's no way to tell whether the bare linefeed is just missing a carriage return or it's some kind of malformed binary data.

If you're seeing stray newline entries in your logs and you're reasonably sure that they're being sent by MTAs or MUAs that intend them to be handled as an end-of-line, use the fixcrio program from the ucspi-tcp package to placate the SMTP daemon. Modify the run script for qmail-smtpd so that it pipes mail through fixcrio, as shown in Example 18-1:

Example SMTP daemon that forgives stray newlines

1. #!/bin/sh
2. limit datasize 3m
3. exec tcpserver \
4. -u000 -g000 -v -p -R \
5. 0 25 \
6. /usr/local/bin/fixcrio | /var/qmail/bin/qmail-smtpd" 2>&1

Line 6 is the modified one, starting up fixcrio and qmail-smtpd. When fixcrio runs, it passes the input and output of qmail-smtpd through pipes so it can add missing carriage returns in front of newlines as needed. In the longer run, see if you can persuade your correspondents to upgrade their SMTP clients to newer, less buggy versions.

How Do I Enable remote access to MySQL database server?

2008-02-28T16:27:00.002+05:30

Step # 1: Login over ssh if server is outside your IDC

First, login over ssh to remote MySQL database server

Step # 2: Enable networking

Once connected you need edit the mysql configuration file my.cfg using text editor such as vi.

* If you are using Debian Linux file is located at /etc/mysql/my.cnf location
* If you are using Red Hat Linux/Fedora Linux file is located at /etc/my.cnf location
* If you are using FreeBSD you need to create a file /var/db/mysql/my.cnf

# vi /etc/my.cnf
Step # 3: Once file open, locate line that read as

[mysqld]
Make sure line skip-networking is commented (or remove line) and add following line
bind-address=YOUR-SERVER-IP

For example, if your MySQL server IP is 65.55.55.2 then entire block should be look like as follows:
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/English
bind-address = 65.55.55.2
# skip-networking
....
..
....Where,

* bind-address : IP address to bind to.
* skip-networking : Don’t listen for TCP/IP connections at all. All interaction with mysqld must be made via Unix sockets. This option is highly recommended for systems where only local requests are allowed. Since you need to allow remote connection this line should removed from file or put it in comment state.

Step# 4 Save and Close the file

Restart your mysql service to take change in effect:# /etc/init.d/mysql restart
Step # 5 Grant access to remote IP address

# mysql -u root -p mysqlGrant access to new database
If you want to add new database called foo for user bar and remote IP 202.54.10.20 then you need to type following commands at mysql> prompt:mysql> CREATE DATABASE foo;
mysql> GRANT ALL ON foo.* TO bar@'202.54.10.20' IDENTIFIED BY 'PASSWORD';
How Do I Grant access to existing database?

Let us assume that you are always making connection from remote IP called 202.54.10.20 for database called webdb for user webadmin, To grant access to this IP address type the following command At mysql> prompt for existing database:mysql> update db set Host='202.54.10.20' where Db='webdb';
mysql> update user set Host='202.54.10.20' where user='webadmin';
Step # 5: Logout of MySQL

Type exit command to logout mysql:mysql> exit
Step # 6: Test it
From remote system type command:

$ mysql -u webadmin –h 65.55.55.2 –p

Where,

* -u webadmin: webadmin is MySQL username
* -h IP or hostname: 65.55.55.2 is MySQL server IP address or hostname (FQDN)
* -p : Prompt for password

You can also use telnet to connect to port 3306 for testing purpose:
$ telnet 65.55.55.2 3306

SMTP and POP with SSL (ucspi-tls-qmail-howto)

2008-02-28T16:21:00.002+05:30

Why You Want To Use UCSPI-TLS

UCSPI-TLS is a protocol for adding "delayed encryption" to Dan Bernstein's Unix Client/Server Program Interface protocol. "Delayed encryption" means a session starts off in plaintext, then a command is issued to turn on encryption, encryption is negotiated, and the session restart. This has become a very common way to handle encryption, because it simplifies client configuration and requires only one TCP port.

These are the goals of UCSPI-TLS:

Simple, UCSPI-compatible use.
Support for both traditional SSL and delayed encryption through STARTTLS or similar.
All SSL/TLS code in one place.
Minimal changes required for each server.
Support for "privilege separation", so that encryption can happen in a low-privilege subprocess.

In particular, I believe the privilege separation feature increases your system's security significantly. It creates a dedicated process to handle each encrypted connection, and this process can change its root directory and switch to a low-privilege user and group. Because of its complexity, OpenSSL has had its share of security bugs. Doing encryption in a low-privilege process ensures that the impact of any security bugs is minimized.

How to install UCSPI-TLS for netqmail-1.05

UCSPI-TLS is currently implemented as patches to existing programs. To install it, first download the programs you want to use, apply the UCSPI-TLS patches, install the programs as you normally would, and finally make the appropriate changes to your configuration files.

Here are more details on doing this with netqmail version 1.05.

Become root, using su or sudo bash. Some of these steps require you to have superuser privileges, and its easier just to get them now than to switch back and forth throughout the process.
Do a base qmail install
Install qmail according to Life with qmail , and make sure everything works with a standard setup.
Install a patched ucspi-ssl+tls
Make sure you have a recent version of OpenSSL installed on your system, along with the files required for development with the library. If you don't already have it installed, this should be available from your OS install CDs, with a name like libssl-dev or openssl-devel. If all else fails, you can download OpenSSL and compile it yourself.
Download ucspi-ssl 0.70, and unpack it. It will create a directory host/superscript.com/net/ucspi-ssl-0.70; cd into this directory. Here's some commands to cut-n-paste:
```
wget http://www.superscript.com/ucspi-ssl/ucspi-ssl-0.70.tar.gz
gunzip -cd ucspi-ssl-0.70.tar.gz |tar xf -
cd host/superscript.com/net/ucspi-ssl-0.70
```

Download ucspi-tls patch to ucspi-ssl (you can also read the ucspitls for ucsp-ssl patch README), and apply the patch:

wget http://www.suspectclass.com/~sgifford/ucspi-tls/files/ucspi-ssl-0.70-ucspitls-0.1.patch
patch -p1 < ucspi-ssl-0.70-ucspitls-0.1.patch

If anything is already using an older copy of sslserver, shut it off now.
Compile and install the patched ucspi-ssl+tls, by running package/compile base then package/install base:
```
package/compile base
package/install base
```
You now have a patched copy of sslserver in /usr/local/bin/sslserver. Congratulations!
Patch qmail
The next step is to patch qmail to add TLS support to its SMTP and POP servers. If you followed Life with qmail religiously, your qmail source code will be in /usr/local/src/netqmail-1.05/netqmail-1.05. cd to that directory, or wherever you have the qmail source:
```
cd /usr/local/src/netqmail-1.05/netqmail-1.05
```

Now download the ucspi-tls patch to netqmail (and read the ucspitls patch for netqmail README), and apply the patch:

wget http://www.suspectclass.com/~sgifford/ucspi-tls/files/netqmail-1.05-ucspitls-0.3.patch
patch -p2 < netqmail-1.05-ucspitls-0.3.patch

If qmail is running, shut it down. On a Life with qmail setup, you do this by running:
```
/var/qmail/bin/qmailctl stop
```
Recompile and re-install qmail by running make setup check:
```
make setup check
```
You now have a copy of qmail-smtpd and qmail-popup with TLS support
Make your certificates
Next you'll need SSL certificates. These certificates contain the encryption keys used to communicate with your servers, and possibly a signature from a trusted authority confirming your server's identity. The first thing you'll need is a place to store them. Let's put them in /var/qmail/ssl:
```
mkdir /var/qmail/ssl
chown root /var/qmail/ssl
chmod 700 /var/qmail/ssl
cd /var/qmail/ssl
```
Create the SSL keys for your system. We'll store the certificate in a file named cert, and the key in key. If you already have certificates you'd like to use, separate out the certificate part (the key part will generally be labeled RSA PRIVATE KEY, and the cert part CERTIFICATE. You can just copy these sections from your certificate file, and paste them into the appropriate files.
You'll have to decide whether you want to use an unsigned certificate or a certificate signed by a Certificate Authority (CA). If you hire a CA to verify your identity and sign your certificate, clients are more likely to accept this certificate without prompting the user. If you use your own unsigned certificate, clients may have to confirm that they trust your certificate.
If you'd like to use a signed certificate, choose a CA, and follow their directions to obtain a signed certificate. Save the certificate in a file called cert, and the key in a file called key. If you just have one file containing both of these, see the beginning of this step for suggestions on how to split it up.
If you'd like to use an unsigned certificate, follow these steps:
1. Make sure openssl is installed on your system.
2. Set a umask that will keep your files protected, by running:
```
umask 077
```
3. Generate a certificate by running:
```
openssl req -new -x509 -keyout key.enc -out cert -days 360
```
  Answer the questions you're asked. Your password is temporary, so it doesn't matter what it is as long as you remember it for a few minutes; you can even write it down. Make sure you use the server's host name, as clients will be configured to connect to it, as the "Common Name" for the certificate. Note that this certificate will expire in 360 days, and you'll have to create a new one before then.
  You'll now have a file called key.enc containing your encrypted key, and a file called cert containing your certificate.
4. Remove the password from your certificate, so the server can start automatically. You can do this by running:
```
openssl rsa -in key.enc -out key
```
  Enter the password you chose above, and you'll have a file called key containing an unencrypted copy of your key.
Create a Diffie-Hellman parameter file. I usually create 1024-bit dhparam files, but I'll admit I don't know exactly what they do. I should probably find out and put it here. You can use the command
```
openssl dhparam -out dhparam 1024
```
to generate a 1024-bit Diffie-Hellman parameter file.
Add a user and group for SSL to drop privileges too; name them both ssl. How to do this depends on your OS and what tools it provides; you can find some examples in the qmail source directory in a file called INSTALL.ids. For example, on Linux you would run:
```
groupadd ssl
useradd -g ssl -d /var/qmail ssl
```
If you can't figure out any OS-specific commands, edit the /etc/group and /etc/passwd files directly, using the file format described in the system manpages for group(5) and passwd(5).

Create a file to set shell variables in /var/qmail/ssl/env. Put these lines in that file:

# Set these three
SSL_USER=ssl
SSL_GROUP=ssl
SSL_DIR=/var/qmail/ssl
# Enable UCSPI-TLS
UCSPITLS=1
# The rest are set based on the above three
SSL_CHROOT="$SSL_DIR"
CERTFILE="$SSL_DIR/cert"
KEYFILE="$SSL_DIR/key"
DHFILE="$SSL_DIR/dhparam"
SSL_UID=`id -u "$SSL_USER"`
if [ $? -ne 0 ]; then echo "No such user '$SSL_USER'" >&2; exit; fi
SSL_GID=`id -g "$SSL_GROUP"`
if [ $? -ne 0 ]; then echo "No such group '$SSL_GROUP'" >&2; exit; fi
# Export the variables used by other scripts
export SSL_CHROOT SSL_UID SSL_GID UCSPITLS CERTFILE KEYFILE DHFILE

Now set your umask back to something more usable, like 022:
```
umask 022
```
Set up qmail-smtpd
Edit the qmail-smtpd run file, in /var/qmail/supervise/qmail-smtpd/run. There are three changes required:
- The top of the file has several variable settings. Below these lines, include the SSL environment variable script we created above, using the shell's "dot" command, typed as a single period:
```
. /var/qmail/ssl/env
```
- On the line that contains softlimit, add 10MB (10,000,000) the number after the -m flag. This allows qmail-smtpd to use the extra memory required for SSL. For example, if it's currently 2000000, you would have for that line:
```
exec /usr/local/bin/softlimit -m 12000000 \
```
- On the line that contains tcpserver, change tcpserver to sslserver -e -n, leaving all of the other flags in place. The line will now look something like:
```
/usr/local/bin/sslserver -e -n -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
```
Restart qmail to use the new configuration:
```
/var/qmail/bin/qmailctl restart
```

Now we'll try a few tests to make sure everything works. If any of these steps fails, double-check the steps above, look for errors in the error logs, and use your common sense to see what's going wrong. Look in the logs to see if there are any errors:

tail /var/log/qmail/smtpd/current

You should see something like this:

@40000000431fb29d10e09c2c sslserver: cafile 32655
@40000000431fb29d10e120fc sslserver: ccafile 32655
@40000000431fb29d10e14bf4 sslserver: cadir 32655 /usr/local/ssl/certs
@40000000431fb29d10e176ec sslserver: cert 32655 /var/qmail/ssl/cert
@40000000431fb29d10e1a9b4 sslserver: key 32655 /var/qmail/ssl/key
@40000000431fb29d10e1d4ac sslserver: param 32655 /var/qmail/ssl/dh 512
@40000000431fb29d10e226b4 sslserver: status: 0/20

Make sure the server is now offering TLS. To test this, you can connect to the server on port 25 and issue an "extended HELO" command (EHLO) and see what extensions the server offers. To do this, start at a shell and type in the lines marked below with *:
```
  * telnet localhost 25
   Trying 127.0.0.1...
   Connected to localhost.
   Escape character is '^]'.
   220 host.example.com ESMTP
 * EHLO there
   250-host.example.com
   250-PIPELINING
   250-8BITMIME
   250 STARTTLS
 * QUIT
   221 host.example.com
```
The line that says STARTTLS after you type EHLO means that the server has TLS configured.
Now test with a client to make sure things basically work. First start watching the logfile for errors:
```
tail --follow=name /var/qmail/supervise/qmail-smtpd/log/main/current
```
Unfortunately the client included with sslserver doesn't support TLS. If you have a copy of stunnel version 3 lying around, you can do something like this:
```
stunnel -r localhost:25 -f -c -n smtp -D debug -P none
```
Otherwise, just skip this test and try the test in the next step.
Finally, try turning on TLS in the mail clients you'll be using and sending a test message. Make sure there are no errors from the client or in the logs, and that the messages arrives successfully.
If this step works, you've set everything up correctly. Congratulations!
Set up qmail-pop3d
If you want to run a POP3 server, make sure you've set up qmail-pop3d as described in Life with qmail, then continue to follow these steps. If you're not running a POP3 server, you can skip this entire section.
Edit the qmail-pop3d run file, in /var/qmail/supervise/qmail-pop3d/run. There are three changes required:
- Near the top of the file, between the #!/bin/sh line and the line that begins with exec, include the SSL environment variable script we created above, using the shell's "dot" command, typed as a single period:
  . /var/qmail/ssl/env
- On the line that contains softlimit, add 10MB (10,000,000) the number after the -m flag. This allows qmail-smtpd to use the extra memory required for SSL. For example, if it's currently 2000000, you would have for that line:
  exec /usr/local/bin/softlimit -m 12000000 \
- On the line that contains tcpserver, change tcpserver to sslserver -e -n, leaving all of the other flags in place. The line will now look something like:
  /usr/local/bin/sslserver -e -n -v -R -H -l 0 0 110 /var/qmail/bin/qmail-popup \
Restart qmail to use the new configuration:
```
/var/qmail/bin/qmailctl restart
```
Now we'll try a few tests to make sure everything works. If any of these steps fails, double-check the steps above, look for errors in the error logs, and use your common sense to see what's going wrong. Look in the logs to see if there are any errors:
```
tail /var/log/qmail/pop3d/current
```
You should see something like this:
```
@40000000431fb29d10e09c2c sslserver: cafile 32655
@40000000431fb29d10e120fc sslserver: ccafile 32655
@40000000431fb29d10e14bf4 sslserver: cadir 32655 /usr/local/ssl/certs
@40000000431fb29d10e176ec sslserver: cert 32655 /var/qmail/ssl/cert
@40000000431fb29d10e1a9b4 sslserver: key 32655 /var/qmail/ssl/key
@40000000431fb29d10e1d4ac sslserver: param 32655 /var/qmail/ssl/dh 512
@40000000431fb29d10e226b4 sslserver: status: 0/20
```
Make sure the server is now offering TLS. To test this, you can connect to the server on port 110, then ask for a list of its capabilities. To do this, start at a shell, and type in the lines marked below with *:
```
  * telnet localhost 110
   Trying 127.0.0.1...
   Connected to localhost.
   Escape character is '^]'.
   +OK
 * CAPA
   +OK capability list follows
   STLS
   .
 * QUIT
   +OK
   Connection closed by foreign host.
```
The STLS line after you type CAPA indicates that the server supports TLS.
Now test with a client to make sure things basically work. First start watching the logfile for errors:
```
tail --follow=name /var/qmail/supervise/qmail-pop3d/log/main/current
```
Unfortunately the client included with sslserver doesn't support TLS. If you have a copy of stunnel version 3 lying around, you can do something like this:
```
stunnel -r localhost:110 -f -c -n pop3 -D debug -P none
```
Otherwise, just skip this test and try the test in the next step.
Finally, try turning on TLS in the mail clients you'll be using and receiving some test messages. Make sure there are no errors from the client or in the logs, and that the messages arrives successfully.
If this step works, you've set everything up correctly. Congratulations!

Optional Step: If desired, set up SSL servers

If you'd like to support SSL (not delayed encryption through TLS), you can still use the modified sslserver, and get the security advantages of chroot and privilege separation.

These steps will help you create an SSL service from a TLS service. The instructions are for qmail-smtpd; they will work for qmail-pop3d if you simply replace smtpd with pop3d everywhere.

Change directory to /var/qmail/supervise:
```
cd /var/qmail/supervise
```
Create a service directory and log directory:
```
mkdir -p qmail-smtpd-ssl/log
```
Copy the run file from the original service, and make it executable:
```
cp qmail-smtpd/run qmail-smtpd-ssl/
chmod 755 qmail-smtpd-ssl/run
```
Edit the run file (qmail-smtpd-ssl/run) in the following ways:
- On the line after . /var/qmail/ssl/env, add unset UCSPITLS
- On the line that contains sslserver line, remove the -n flag.
- On the next line, which will contain something like 0 smtp, change smtp to smtps; that tells sslserver to listen on the appropriate port for the SSL version of this service.

Set up a logging directory for this new service:

mkdir /var/log/qmail/smtpd-ssl
chown qmaill /var/log/qmail/smtpd-ssl

Set up the logging program for this new service, by creating a file in qmail-smtpd-ssl/log/run with these contents:
```
#!/bin/sh
exec /usr/local/bin/setuidgid qmaill /usr/local/bin/multilog t \
      /var/log/qmail/smtpd-ssl
```
Make sure the script is executable:
```
chmod 755 qmail-smtpd-ssl/log/run
```
Link the new service into the /service directory, to have it start automatically on boot:
```
ln -s /var/qmail/supervise/qmail-smtpd-ssl /service
```

Add the following to qmailctl's "start" section:

if svok /service/qmail-smtpd-ssl ; then
 svc -u /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log
else
 echo qmail-smtpd-ssl supervise not running
fi

Add the following to qmailctl's "stop" section:

echo "  qmail-smtpd-ssl"
svc -d /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log

Add the following to qmailctl's "stat" section:

svstat /service/qmail-smtpd-ssl
svstat /service/qmail-smtpd-ssl/log

Add the following to qmailctl's "pause" section:

echo "Pausing qmail-smtpd-ssl"
svc -p /service/qmail-smtpd-ssl

Add the following to qmailctl's "cont" section:

echo "Continuing qmail-smtpd-ssl"
svc -c /service/qmail-smtpd-ssl

Add the following to qmailctl's "restart" section:

echo "* Restarting qmail-smtpd-ssl."
svc -t /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log

To test, use an SSL-enabled mail client, or use stunnel version 3:
```
stunnel -r localhost:smtps -f -c -D debug -P none
```

Optional Step: If desired, set up other servers

If you have any other TLS or SSL servers you'd like to set up, such as an IMAP server, you should be able to use the above steps as a template to get you started.

Required Step: Sleep better at night

Knowing your TLS- and SSL-protected mail server is secure.

Thanks

Rbldns How to

2008-02-28T16:19:00.001+05:30

What is this document?

I wrote this HOWTO so that others can have one place to stop and see how to setup their own private RBL list. The information is out there, but scattered all over the place.

The color convention I use is to have the headlines in gray and the code / shell scripts in yellow

What is rbldns?

Rbldns is a suite of programs in conjunction with dnscache to provide RBL service. In a nutshell IPs put into the rbldns data file show up as 127.0.0.1 responses to a specially crafted DNS query.

The IP address go into the file in the normal readable way, ie 192.168.0.1, and you query for them by reversing the address, like 1.0.168.192.example.com

How do I use rbldns?

Normally you don't use it directly, you have a program query an RBL server and based on the result of the query it either runs or doesn't run the program following it.

For example with Qmail you'll change your qmail-smtpd/run file from
tcpserver 0 smtp /var/qmail/bin/qmail-smtpd
to
tcpserver 0 smtp /usr/local/bin/rblsmtpd -a whitelist.example.com -r blacklist.example.com -r relays.ordb.org /var/qmail/bin/qmail-smtpd

According to the rblsmtpd man page, the -r and -a switches mean:

Switch	If successful, do this
-r	Quit
-a	Don't do other lookups, and continue on

In this case the steps are:
1. Do a lookup against whitelist.example.com, if sucessful then skip the other lookups and go on to accepting the smtp request
2. Do a lookup against blacklist.example.com, if that's successful then quit out printing a "RBL denied" message to the client.
3. Same as #2 but against relays.ordb.org

How do I make my own RBL?

You should know how to setup Dnscache before trying to do this. The steps are pretty easy, but its helpful if you know your way around

Setting up the rbldns is pretty easy, just follow the directions on the RBLdns-conf page:

 rbldns-conf rbldns dnslog /etc/rbldns/black 127.0.0.2 rbl.example.com
rbldns-conf rbldns dnslog /etc/rbldns/white 127.0.0.3 whitelist.example.com

This says to setup a RBL on 127.0.0.2 that answers queries for *.rbl.example.com, and and RBL on 127.0.0.3 that answers for *.whitelist.example.com

Note: the 'rbl' and 'whitelist' names are arbitrary. RBLdns doesn't know if this is a 'good' or a 'bad' list of addresses -- it just knows that if someone asks for 1.0.168.192.bad.example.com on IP 127.0.0.2 it should answer with a 127.0.0.1 if it has it, or nothing if it doesn't.

Start these services up like all of DJB's other daemons:

 ln -s /etc/rbldns/black /service
ln -s /etc/rbldns/white /service

To see if they are running, do a

 lsof -n -i:53

And you should see listeners on 127.0.0.2 and 127.0.0.3

Why would I want to use a local RBL?

Why shouldn't you just relay on publically available RBLs? For a couple of reasons:
1. You know a couple of IPs are good and you don't want to waste bandwith by checking them against an RBL on the internet
2. Same thing for bad ones
3. You know your friend has a DSL line at home and an RBL blocks all home DSL users, you want to continue using the service but want to accept his mail.
4. You want to keep a 2nd source of information about who is sending mail to your server

So part of this is being a good netizen. If you know you're getting a lot of email from a handful of IPs, you should put those in your own whitelist so that you're not continously asking the public RBLs if they are okay

How to resolve *.(whitelist|rbl).example.com -- DNScache way

Now that we have the RBL service ready to go we have to be able to query it. Like we told dnscache to send all queries for *.example.com to the local tinydns IP address, we're going to do that for the RBL service.

This is the part that took me a while to think about: how do I serve both *.example.com and *.rbl.example.com? Well the same way you've already done it: through a dnscache setting.

Add two entries to your dnscache "forwarding" information:

 cd /service/dnscache/root/servers
echo 127.0.0.2 > rbl.example.com
echo 127.0.0.3 > whitelist.example.com
svc -h /service/dnscache

This tells dnscache that when it gets a query for *.whitelist.example.com to send it off to the server on 127.0.0.3, *.rbl.example.com queries should go to 127.0.0.2. If you're setup TinyDns this should be familiar

How to resolve *.(whitelist|rbl).example.com -- TinyDns way

This might be a little cleaner if you're already serving up *.example.com results. You put in your tinydns file pointers to the (rbl|whitelist) lists:

 cd /service/tinydns/root/
echo "&rbl.example.com:127.0.0.2:a" >> data
echo "&whitelist.example.com:127.0.0.3:a" >> data
make
svc -h /service/tinydns

So now when your tinydns server is queried for 1.0.168.192.rbl.example.com it will look at its file and say "That's handled by 127.0.0.2 and I'll ask it"

How do I add entries to my RBL?

Scenerio: a spammer is continously sending us email from 192.168.5.100 while our friend is sending from 192.168.6.56. We would like to avoid using public RBL servers for these two addresses.

Step 1: add the spammer to your spam list:

 cd /service/rbl/black/root
echo 192.168.5.100 >> data
make

Step 2: Add your friend to your good list:

 cd /service/rbl/white/root
echo 192.168.6.56 >> data
make

Now query to see if they are in there:

 dnsip 100.5.168.192.rbl.example.com
dnsip 100.5.168.192.whitelist.example.com
dnsip 56.6.168.192.rbl.example.com
dnsip 56.6.168.192.whitelist.example.com

Notice I queried both sets of databases to make sure we didn't screw up who is a spamer versus who is our friend.

If you don't get a 127.0.0.1 result from a query when you think you should, HUP the appropriate daemon:

 svc -h /service/rbl/black
svc -h /service/rbl/white

How do I now use an RBL service?

Now that you have the local RBLdns server(s) working through simple queries, modify your other scripts to use them. With qmail you do that like this:

exec /usr/local/bin/tcpserver -R -v -p \
  -x /var/qmail/control/tcp.smtp.cdb \
  -u $QMAILUID -g $QMAILGID \
  mail.example.com smtp \
  /usr/local/bin/rblsmtpd \
  -a whitelist.example.com \
  -r rbl.example.com \
  -r relays.ordb.org \
  -r bl.spamcop.net \
  /var/qmail/bin/qmail-smtpd 2>&1

This is what your implementation will probably look like: an initial test against your goodlist, if that passes the continue on to the qmail-smtpd part. If that fails, then test against your local bad list. If that is sucessful then quit out. If not, continue on to your 2nd and 3rd RBLs.

How do I test my RBL?

You test an RBL by sending it a query. If it is in the database then it should return with a 127.0.0.x (1 or 2) answer. If it isn't in there, then nothing comes back.

The trick is to remember to reverse your IP address, so that 192.168.0.1 becomes 1.0.168.192.myrbldomain.com

Here's a simple program to check names fed on stdin:

 #!/usr/bin/perl

use Net::DNS;
use strict;
my ($res, @rbls);

@rbls = qw (rbl.example.com whitelist.example.com);

$res  = Net::DNS::Resolver->new;
while (<>) {
 my ($rev, $ip);
 chomp;
 $ip = $_;
 $rev = (join ".", reverse (split /\./, $ip) );  # ie 4.3.2.1.
 foreach my $rbl (@rbls) {
  my $query = $res->search("$rev.$rbl");
  next unless ($query);
  my $hit;
  foreach my $rr ($query->answer) {
   next unless $rr->type eq "A";
   $hit = $rr->address;
   last;
  }
  next unless $hit;
  printf "%-20s %-20s $hit\n", $ip, $rbl;
 }
}

How do I contact you?

If you have any questions or comments, or just want to say thanks, drop me a line at anand.shah@coolaquarius.com

Easy steps to preventing spam

2007-09-19T19:00:00.000+05:30

Easy steps to preventing spam

The following 10 steps offer a very common sense (though often overlooked) approach to avoiding spam. If you follow these guidelines, you’ll notice a drastic decrease in the number of annoying messages that make it into your inbox…

01. Give your primary email address to friends and family only. Give a different address to others on the Internet. Although this second address will be more likely to receive unwanted emails, it is more disposable and can allow you to better control the emails you receive. Your ISP almost certainly offers multiple email addresses, and Yahoo, Microsoft and Google all offer free web based email accounts. These are great to use as disposable addresses.

02. Make sure your email address is difficult to guess. Don’t use just a common name or common word(s). Spammers use software programs to generate random email addresses based on common names and words in the dictionary.
It is NOT uncommon for a brand new email address to start receiving spam almost immediately after it has been created. This happens because even though the address is new, spammers will take a previously used user name (eg. jdoe@whatever.com) and apply it to a different domain (eg. jdoe@another_domain.com).
In addition, common email addresses may have been used previously and may still be on old mailing lists.

03. Do not post your primary email address in newsgroups, bulletin boards or chat rooms. Spammers use software programs, often referred to as spiders or bots, to search for and harvest email addresses from public forums. To prevent this, use a secondary email address or alter the primary address so that it is not deliverable in that format. For example, if your email address is jdoe@whatever.com, you could post it as jdoe@TAKE-THIS-OUT.whatever.com or “jdoe at whatever dot com”.

04. Do not post your primary email address on a Web site. Spiders also scan Web sites for email addresses. You can alter your email address to help protect it but remember that email harvesting software can read HTML code, so be sure to remove the “mailto:” tag.

05. Do not reply to unsolicited emails. If the email does not appear to be from a trustworthy or legitimate source, delete it without replying. A federal anti-spam law called The Can Spam Act, went into effect January 1, 2004, requiring a functioning “opt out” link or a legitimate “reply to unsubscribe” email address. Some unscrupulous spammers have ignored this law and continue to trick recipients into unwittingly responding to a fake “opt out” link, which actually verifies their email address as a valid one. Therefore, it is still strongly recommended that recipients of unsolicited email carefully consider whether an “opt out” or “reply to unsubscribe” seems legitimate and act accordingly.

06. Consider using an alternate email address when signing up for services, filling out forms or taking surveys on the Internet. Read the privacy policy of these sites. Keep in mind, if the service is “free” they often need to generate revenue in some manner and advertising is often used to do this.

07. When signing up for a mailing list, read the terms and policies. Signing up should result in wanted or solicited email, but the list provider should disclose whether signing up will result in the sale or trade of your email address to other parties.

08. Let friends and family know that you do not wish to have them share your email address.

09. Keep your PCs anti-virus software up-to-date and install a firewall. Unprotected high-speed Internet connections are vulnerable to infection by viruses that are programmed to open gateways, also known as proxies, to relay spam. By not keeping your PC secure, you may unwittingly be a courier for spam. There’s a good chance your ISP offers free or discounted security software - this article can help you figure out if yours does.

10. Check “sent mail” folders for suspicious messages. Take responsibility for your PC by checking your “sent mail” folder regularly to ensure that all sent mail is really being sent by you and not by a spammer using an open gateway (proxy) on your computer.

Anand Shah

Block IP how to in Windows IIS and Apache

2007-09-19T18:53:00.000+05:30

Block IP How To

The following tutorial explainss how you can block IP addresses from viewing your website in Windows IIS and Linux/Unix Apache. Blocking IP addresses prevents users from seeing your website during construction, limits access to specific users, or to blocks users attempting to connect to your site maliciously.

Block IP how to in Windows IIS

To block an IP address from viewing your website, please follow these steps:
1. Login to your server through Terminal Services or Remote Desktop Connection. 2. Click Start, select Programs, and then click Administrative Tools. For IIS 5.0 click Internet Services Manager. For IIS 6.0 click Internet Information Services.3. In the left column you will see the Server Name. In IIS 5.0, expand the Server Name to find the domain name. In IIS 6.0, expand the Server Name and then Web Sites to find the domain name.4. Right-click on the domain name and select Properties.5. On the Directory Security Tab under IP Address and Domain Name Restrictions click Edit.You have two options, you can grant access to all computers and restrict individual IP's or you can deny access to all computers and grant access to specific IP's. By default, all users will be granted access to your site except the IP addresses you specify.To add an IP address to the restriction list, please follow these steps: 1. Click Add.2. Select the type: Single Computer - add a single IP Address. Group of Computers - add a block IP Address using the starting IP address of the block and the appropriate subnet mask. Domain Name - add a domain name (this method is not recommended). 3. Click Ok.

Block IP how to in Linux/Unix Apache

You can also block (or admit) users based on their incoming IP address: RewriteEngine on RewriteMap block dbm:/www/conf/my.block RewriteCond ${block:%{REMOTE_ADDR}OK} !^OK$ RewriteRule ^/.* http://%{REMOTE_ADDR}/ [L]
You create my.block.db from a file (named blocklist) that looks like this: xxx.xxx.xxx.xxx block
and piping it to./db_create my.block.db < blocklist
You can add additional entries on the fly: echo "xxx.xxx.xxx.xxy block" ./db_create

Anand Shah

Blocking Spam with Sendmail

2007-09-19T18:50:00.000+05:30

Like a great many people on the net, I found myself increasingly annoyed by the rising tide of spam. By which I do not mean the delectible Hormel canned meat product (which the little lady especially likes fried in a fried egg sandwich), but the phenomenon of unsolicited commercial email.

Spam is unlike regular junk mail from the post office for several reasons:

Normal junk mail costs money to send. Spam does not. The cost is born by the recipients and by the servers in between. While normal junk mail is a revenue source for the post office and actually helps to pay for delivering regular mail, spam pays for nothing. Spam can hence be viewed as theft of services.

Spam has detrimental effects. Historically Internet mail servers have passed along email between third parties as a gesture of friendship and good will. Much like tossing your neighbor's paper over the fence if the paperboy misses his throw. This is happening less and less because spammers take advantage of it.

Spam reduces the utility of email. People become discouraged about checking their mailboxes if they are always cluttered with spam.

Enough ranting. I decided to do something. Being the sort who favors technical fixes over legal ones, I started doing some research on the web, ordered a copy of the Bat book, and spent some time reading my sendmail configuration and scratching my head. I present here the result.
First, if you're not using sendmail I can't help you. Second, you need the latest version of sendmail or these tricks won't work. And finally, we had several conditions that had to be met:

We wanted to be able to block spam by domain name, network number, or by specific address for maximum flexibility.

We needed to be able to allow spam to selected mailboxes for customers who do not want spam blocked. I may disagree with them, but they are paying for a service and it is, after all, their mail. Spam blocking had to be a value added service that could be turned off.

We host a number of "virtual domains" and needed to be able to route email for them to the proper mailboxes. We had already been doing that, but it was a factor that had to be considered in our antispam measures so that spam could be blocked or not as desired by the mailbox owners.

We wanted to stop "third party relay" going through our mail server while allowing for exceptions for customers with their own domains and mail servers or for roaming customers.

I think I have come up with a set of sendmail rules which accomplish this.
First, we need to add a few entries in the local configuration section:

LOCAL_CONFIG

Fw/etc/vdomain.cw
Kvdomain hash /etc/vdomain

# list of people who like spam
F{fools} /etc/WantSpam

# list of known spammers
Kjunk hash -a@JUNK /etc/spammers

# List of network addresses we will relay for
F{LocalIP} /etc/LocalIP

# List of domains we will relay to
F{RelayTo} /etc/RelayTo

Click on the filename of any of these files for an explanation of its purpose and contents.
Now we get to the rules themselves. First, an entry must be added to your local rule zero, like so:

LOCAL_RULE_0
R$* $: $>vmap $1

Not very interesting, is it? It just calls another rule set, named "vmap", which handles virtual domain address mapping. Note: I don't know what the "right way" is to do these things, but it works to just list all the rest of these rulesets right under LOCAL_RULE_0, so that's what I do. Here then is the "vmap" ruleset:
Svmap
R$+ < @ $+ . > $: $1 < @ $2 > .
R$+ < @ $+ > $* $: $(vdomain $1@$2 $: $1 < @ $2 > $3 $)
R$+ < @ $+ > $* $: $(vdomain $2 $: $1 < @ $2 > $3 $)
R$+ < @ $+ > . $: $1 < @ $2 . >

I made this a separate ruleset since I do it again in the rest of the rules, as you will see. I have obscure reasons for not just calling local rule zero as needed.
Next I define a "junk" ruleset to look up a domain name or email address in the /etc/spammers.db database:

Sjunk
R$* $: $(junk $1$) look for host in spammer list
R$+@JUNK $@ $1@JUNK return message if found
R@JUNK $@ Spam refused @JUNK generic message
R$-.$+ $: $1 . $>junk $2 retry skipping lead subdomain
R$-.$+@JUNK $@ $2@JUNK return message if found

Next, a "junkIP" ruleset to look up an IP address or network number in the /etc/spammers.db database:
SjunkIP
R$* $: $(junk $1$) look for host in spammer list
R$+@JUNK $@ $1@JUNK return message if found
R@JUNK $@ Spam refused @JUNK generic message
R$+.$- $: $2 . $>junkIP $1 retry without trailing number
R$-.$+@JUNK $@ $2@JUNK return message if found
R$-.$+ $@ $2.$1 fix order if not spammer

Now for the heart of it, the "check_rcpt" ruleset. Spam blocking is more often done in the "check_mail" ruleset, but we can't do it that way since we need to check the recipient to see if they want spam. Hence, this ruleset gets a bit long.
Scheck_rcpt
R$* $: $>vmap $>3 $1 normalize address

# Refuse to relay mail between nonlocal systems
R$* $: $(dequote "" $&{client_addr} $) $ $1
R0 $ $* $@ ok no client addr: directly invoked
R$={LocalIP}$* $ $* $@ ok from here
R$* $ $* $: $2 not from local, check recipient
R$*<@$=w.>$* $>3 $1 $3 remove our aliases, maybe repeatedly
R$*<@$*$={RelayTo}.>$* $>3 $1 $4 remove domains we relay to
# still something left?
R$*<@$+>$* $#error $@ 5.5.4 $: "554 we do not relay from " $&{client_name} " to " $1@$2$3

# Allow mail to fools who like spam, and otherwise block spammers
R$={fools} $@ ok recipient listed as wanting spam

# Block by host or domain name
R$* $: $(dequote "" $&{client_name} $)
R$* $: $>junk $1
R$*@JUNK $#error $@ 5.5.4 $: "554 " $1 ": " $&{client_name}

# Block by network or host IP address
R$* $: $(dequote "" $&{client_addr} $)
R$* $: $>junkIP $1
R$*@JUNK $#error $@ 5.5.4 $: "554 " $1 ": " $&{client_addr}

# Block by specific email address
R$* $: $(dequote "" $&f $)
R$* $: $>junk $1
R$*@JUNK $#error $@ 5.5.4 $: "554 " $1 ": " $&f
R$* @ $* $: $1 @ $>junk $2
R$* @ $*@JUNK $#error $@ 5.5.4 $: "554 " $2 ": " $&f

# Block mail from invalid addresses
R$* $: $>3 $1 make domain canonical
R$* < @ $+ .> $* $@ ok name resolved ok
# Killer case -- single token domain
R$* < @ $- > $* $#error $@ 5.5.1 $: "551 Invalid host name: " $2
# Delay case -- domain doesn't resolve
R$* < @ $+ > $* $#error $@ 4.5.1 $: "451 Unknown domain: " $2

And that's it. If you'd like, you can download a text version of this for easier editing.
Oh, one last thing. The rejection messages all get logged in /var/log/maillog (at least on our system). Here's a PERL script for maillog.scan that gives us a nightly report of spam blocks:

#!/usr/bin/perl

while($lt;$gt;) {
if(/rejection:.*\.\.\. ? ?(.*)/) {
$spam{$1} += 1;
}
}

print "\nSpam blocks:\n\n";

foreach $msg (sort keys %spam) {
printf "%5d %s\n", $spam{$msg}, $msg;
}

print "\n";

Anand Shah