Thursday, May 22, 2008

PERL ONE LINERS

Level: Introductory

This article, as regular readers may have guessed, is the sequel to "One-liners 101," which appeared in a previous installment of "Cultured Perl". The earlier article is an absolute requirement for understanding the material here, so please take a look at it before you continue.

The goal of this article, as with its predecessor, is to show legible and reusable code, not necessarily the shortest or most efficient version of a program. With that in mind, let's get to the code!

Awk is commonly used for basic tasks such as breaking up text into fields; Perl excels at text manipulation by design. Thus, we come to our first one-liner, intended to add two columns in the text input to the script.

Listing 1. Like awk?

# add first and penultimate columns
# NOTE the equivalent awk script:
# awk '{i = NF - 1; print $1 + $i}'
perl -lane 'print $F[0] + $F[-2]'

So what does it do? The magic is in the switches. The -n and -a switches make the script a wrapper around input that splits the input on whitespace into the @F array; the -e switch adds an extra statement into the wrapper. The code of interest actually produced is:

Listing 2: The full Monty

while (<>)
{
@F = split(' ');
print $F[0] + $F[-2]; # offset -2 means "2nd to last element of the array"
}

Another common task is to print the contents of a file between two markers or between two line numbers.

Listing 3: Printing a range of lines

# 1. just lines 15 to 17
perl -ne 'print if 15 .. 17'

# 2. just lines NOT between line 10 and 20
perl -ne 'print unless 10 .. 20'

# 3. lines between START and END
perl -ne 'print if /^START$/ .. /^END$/'

# 4. lines NOT between START and END
perl -ne 'print unless /^START$/ .. /^END$/'

A problem with the first one-liner in Listing 3 is that it will go through the whole file, even if the necessary range has already been covered. The third one-liner does not have that problem, because it will print all the lines between the START and END markers. If there are eight sets of START/END markers, the third one-liner will print the lines inside all eight sets.

Preventing the inefficiency of the first one-liner is easy: just use the $. variable, which tells you the current line. Start printing if $. is over 15 and exit if $. is greater than 17.

Listing 4: Printing a numeric range of lines more efficiently

# just lines 15 to 17, efficiently
perl -ne 'print if $. >= 15; exit if $. >= 17;'

Enough printing, let's do some editing. Needless to say, if you are experimenting with one-liners, especially ones intended to modify data, you should keep backups. You wouldn't be the first programmer to think a minor modification couldn't possibly make a difference to a one-liner program; just don't make that assumption while editing the Sendmail configuration or your mailbox.

Listing 5: In-place editing

# 1. in-place edit of *.c files changing all foo to bar
perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

# 2. delete first 10 lines
perl -i.old -ne 'print unless 1 .. 10' foo.txt

# 3. change all the isolated oldvar occurrences to newvar
perl -i.old -pe 's{\boldvar\b}{newvar}g' *.[chy]

# 4. increment all numbers found in these files
perl -i.tiny -pe 's/(\d+)/ 1 + $1 /ge' file1 file2 ....

# 5. delete all but lines between START and END
perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

# 6. binary edit (careful!)
perl -i.bak -pe 's/Mozilla/Slopoke/g' /usr/local/bin/netscape

Why does 1 .. 10 specify line numbers 1 through 10? Read the "perldoc perlop" manual page. Basically, the .. operator iterates through a range. Thus, the script does not count 10 lines, it counts 10 iterations of the loop generated by the -n switch (see "perldoc perlrun" and Listing 2 for an example of that loop).

The magic of the -i switch is that it replaces each file in @ARGV with the version produced by the script's output on that file. Thus, the -i switch makes Perl into an editing text filter. Do not forget to use the backup option to the -i switch. Following the i with an extension will make a backup of the edited file using that extension.

Note how the -p and -n switch are used. The -n switch is used when you want explicitly to print out data. The -p switch implicitly inserts a print $_ statement in the loop produced by the -n switch. Thus, the -p switch is better for full processing of a file, while the -n switch is better for selective file processing, where only specific data needs to be printed.

Examples of in-place editing can also be found in the "One-liners 101" article.

Reversing the contents of a file is not a common task, but the following one-liners show than the -n and -p switches are not always the best choice when processing an entire file.

Listing 6: Reversal of files' fortunes

# 1. command-line that reverses the whole input by lines
# (printing each line in reverse order)
perl -e 'print reverse <>' file1 file2 file3 ....

# 2. command-line that shows each line with its characters backwards
perl -nle 'print scalar reverse $_' file1 file2 file3 ....

# 3. find palindromes in the /usr/dict/words dictionary file
perl -lne '$_ = lc $_; print if $_ eq reverse' /usr/dict/words

# 4. command-line that reverses all the bytes in a file
perl -0777e 'print scalar reverse <>' f1 f2 f3 ...

# 5. command-line that reverses each paragraph in the file but prints
# them in order
perl -00 -e 'print reverse <>' file1 file2 file3 ....

The -0 (zero) flag is very useful if you want to read a full paragraph or a full file into a single string. (It also works with any character number, so you can use a special character as a marker.) Be careful when reading a full file in one command (-0777), because a large file will use up all your memory. If you need to read the contents of a file backwards (for instance, to analyze a log in reverse order), use the CPAN module File::ReadBackwards. Also see "One-liners 101," which shows an example of log analysis with File::ReadBackwards.

Note the similarity between the first and second scripts in Listing 6. The first one, however, is completely different from the second one. The difference lies in using <> in scalar context (as -n does in the second script) or list context (as the first script does).

The third script, the palindrome detector, did not originally have the $_ = lc $_; segment. I added that to catch those palindromes like "Bob" that are not the same backwards.

My addition can be written as $_ = lc; as well, but explicitly stating the subject of the lc() function makes the one-liner more legible, in my opinion.

Listing 7: Rewrite with a random number

# replace string XYZ with a random number less than 611 in these files
perl -i.bak -pe "s/XYZ/int rand(611)/e" f1 f2 f3

This is a filter that replaces XYZ with a random number less than 611 (that number is arbitrarily chosen). Remember the rand() function returns a random number between 0 and its argument.

Note that XYZ will be replaced by a different random number every time, because the substitution evaluates "int rand(611)" every time.

Listing 8: Revealing the files' base nature

# 1. Run basename on contents of file
perl -pe "s@.*/@@gio" INDEX

# 2. Run dirname on contents of file
perl -pe 's@^(.*/)[^/]+@$1\n@' INDEX

# 3. Run basename on contents of file
perl -MFile::Basename -ne 'print basename $_' INDEX

# 4. Run dirname on contents of file
perl -MFile::Basename -ne 'print dirname $_' INDEX

One-liners 1 and 2 came from Paul, while 3 and 4 were my rewrites of them with the File::Basename module. Their purpose is simple, but any system administrator will find these one-liners useful.

Listing 9: Moving or renaming, it's all the same in UNIX

# 1. write command to mv dirs XYZ_asd to Asd
# (you may have to preface each '!' with a '\' depending on your shell)
ls | perl -pe 's!([^_]+)_(.)(.*)!mv $1_$2$3 \u$2\E$3!gio'

# 2. Write a shell script to move input from xyz to Xyz
ls | perl -ne 'chop; printf "mv $_ %s\n", ucfirst $_;'

For regular users or system administrators, renaming files based on a pattern is a very common task. The scripts above will do two kinds of job: either remove the file name portion up to the _ character, or change each filename so that its first letter is uppercased according to the Perl ucfirst() function.

There is a UNIX utility called "mmv" by Vladimir Lanin that may also be of interest. It allows you to rename files based on simple patterns, and it's surprisingly powerful. See the Resources section for a link to this utility.

Some of mine

The following is not a one-liner, but it's a pretty useful script that started as a one-liner. It is similar to Listing 7 in that it replaces a fixed string, but the trick is that the replacement itself for the fixed string becomes the fixed string the next time.

The idea came from a newsgroup posting a long time ago, but I haven't been able to find original version. The script is useful in case you need to replace one IP address with another in all your system files -- for instance, if your default router has changed. The script includes $0 (in UNIX, usually the name of the script) in the list of files to rewrite.

As a one-liner it ultimately proved too complex, and the messages regarding what is about to be executed are necessary when system files are going to be modified.

Listing 10: Replace one IP address with another one

#!/usr/bin/perl -w

use Regexp::Common qw/net/; # provides the regular expressions for IP matching

my $replacement = shift @ARGV; # get the new IP address

die "You must provide $0 with a replacement string for the IP 111.111.111.111"
unless $replacement;

# we require that $replacement be JUST a valid IP address
die "Invalid IP address provided: [$replacement]"
unless $replacement =~ m/^$RE{net}{IPv4}$/;

# replace the string in each file
foreach my $file ($0, qw[/etc/hosts /etc/defaultrouter /etc/ethers], @ARGV)
{
# note that we know $replacement is a valid IP address, so this is
# not a dangerous invocation
my $command = "perl -p -i.bak -e 's/111.111.111.111/$replacement/g' $file";

print "Executing [$command]\n";
system($command);
}

Note the use of the Regexp::Common module, an indispensable resource for any Perl programmer today. Without Regexp::Common, you will be wasting a lot of time trying to match a number or other common patterns manually, and you're likely to get it wrong.

Handy One Liners for SED COmmand

Handy one-liners for SED

HANDY ONE-LINERS FOR SED (Unix stream editor)

Latest version of this file is usually at:
http://www.student.northpark.edu/pemente/sed/sed1line.txt
http://www.cornerstonemag.com/sed/sed1line.txt

FILE SPACING:

# double space a file
sed G

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed '/^$/d;G'

# triple space a file
sed 'G;G'

# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'

NUMBERING:

# number each line of a file (simple left alignment). Using a tab (see
# note on '\t' at end of file) instead of space will preserve margins.
sed = filename | sed 'N;s/\n/\t/'

# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *$.\{6,\}$\n/\1 /'

# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'

# count lines (emulates "wc -l")
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$//" # method 1
sed -n p # method 2

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of sed. Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file

# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file

# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'

# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'

# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space

# center all text in the middle of 79-column width. In method 1,
# spaces at the beginning of the line are significant, and trailing
# spaces are appended at the end of the line. In method 2, spaces at
# the beginning of the line are discarded in centering the line, and
# no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/$ *$\1/\1/' # method 2

# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/$.*$foo$.*foo$/\1bar\2/' # replace the next-to-last case
sed 's/$.*$foo/\1bar/' # replace only the last case

# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'

# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only

# reverse order of lines (emulates "tac")
# bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2

# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/$.$$.*\n$/&\2\1/;//D;s/.//'

# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'

# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'

# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/$.*[0-9]$$[0-9]\{3\}$/\1,\2/;ta' # other seds

# add commas to numbers with decimal points and minus signs (GNU sed)
gsed ':a;s/$^\|[^0-9.]$$[0-9]\+$$[0-9]\{3\}$/\1\2,\3/g;ta'

# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds

SELECTIVE PRINTING OF CERTAIN LINES:

# print first 10 lines of file (emulates behavior of "head")
sed 10q

# print first line of file (emulates "head -1")
sed q

# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'

# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'

# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2

# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2

# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax

# print the line immediately before a regexp, but not the line
# containing the regexp
sed -n '/regexp/{g;1!p;};h'

# print the line immediately after a regexp, but not the line
# containing the regexp
sed -n '/regexp/{n;p;}'

# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'

# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'

# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed only

# print paragraph if it contains AAA (blank lines separate paragraphs)
# HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only

# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'

# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax

# print section of file from regular expression to end of file
sed -n '/regexp/,$p'

# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2

# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files

# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds

# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'

# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^$.*$\n\1$/!P; D'

# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^$[ -~]*\n$.*\n\1/d; s/\n//; h; P'

# delete the first 10 lines of a file
sed '1,10d'

# delete the last line of a file
sed '$d'

# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'

# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2

# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds

# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2

# delete all CONSECUTIVE blank lines from file except the first; also
# deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF

# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'

# delete all leading blank lines at top of file
sed '/./,$!d'

# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02*

# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:

# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed v1.5

# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line

# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line

# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'

# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'

# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'

# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2

# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^$.*$\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:

cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk

For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult "sed &
awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,
1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty
and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst
distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power
of sed, one must understand "regular expressions." For this, see
"Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997).
The manual ("man") pages on Unix systems may be helpful (try "man
sed", "man regexp", or the subsection on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach sed use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.

QUOTING SYNTAX: The preceding examples use single quotes ('...')
instead of double quotes ("...") to enclose editing commands, since
sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes
(`...`), which are expanded by the shell if they are enclosed in
double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., \!) to
properly run the examples listed above, even within single quotes.
Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.

USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
However, most versions of sed do not recognize the '\t' abbreviation,
so when typing these scripts from the command line, you should press
the TAB key instead. '\t' is supported as a regular expression
metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.

VERSIONS OF SED: Versions of sed do differ, and some slight syntax
variation is to be expected. In particular, most do not support the
use of labels (:name) or branch instructions (b,t) within editing
commands, except at the end of those commands. We have used the syntax
which will be portable to most users of sed, even though the popular
GNU versions of sed allow a more succinct syntax. When the reader sees
a fairly long command such as this:

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d

it is heartening to know that GNU sed will let you reduce it to:

sed '/AAA/b;/BBB/b;/CCC/b;d' # or even
sed '/AAA\|BBB\|CCC/b;d'

In addition, remember that while many versions of sed accept a command
like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which
contains space before the 's'. Omit the space when typing the command.

OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the "find" expression is specified before
giving the "s/.../.../" instruction. Thus:

sed 's/foo/bar/g' filename # standard replace command
sed '/foo/ s/foo/bar/g' filename # executes more quickly
sed '/foo/ s//bar/g' filename # shorthand sed syntax

On line selection or deletion in which you only need to output lines
from the first part of the file, a "quit" command (q) in the script
will drastically reduce processing time for large files. Thus:

sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster

Anand Shah
Rediff.com

Wednesday, March 26, 2008

How does ssl work

Internet communication typically runs through multiple program layers on a server before getting to the requested data such as a web page or cgi scripts.

The outer layer is the first to be hit by the request. This is the high level protocols such as HTTP (web server), IMAP (mail server), and FTP (file transfer).

Determining which outer layer protocol will handle the request depends on the type of request made by the client. This high level protocol then processes the request through the Secure Sockets Layer. If the request is for a non-secure connection it passes through to the TCP/IP layer and the server application or data.

If the client requested a secure connection the ssl layer initiates a handshake to begin the secure communication process. Depending on the SSL setup on the server, it may require that a secure connection be made before allowing communication to pass through to the TCP/IP layer in which case a non-secure request will send back an error asking for them to retry securely (or simply deny the non-secure connection).

This is a good time to answer a question we periodically get: "what does ssl encrypt?"

This question is usually geared toward whether or not the path and query string is encrypted in an HTTPS "get" request (this is where form field responses or program variables are tagged on to the end of the url). These fields are stripped off of the URL when creating the routing information in the https packaging process by the browser and are included in the encrypted data block.

The page data (form, text, and query string) are passed in the encrypted block after the encryption methods are determined and the handshake completes.

A related issue that frequently comes up is whether or not form data is transmited with encryption if the blank form is displayed without https. If the form "action" is set to use https then the ssl handshake will take place before the data is sent. Whether or not the original form is displayed using https has little to do with the form submission unless the form action uses a relative path, in which case the default will be to use the protocol that was used to display the form.

PHP SESSION HANDLING EXAMPLE

The below code is a very simple example of Session.
Please note that the "session_start" statement *must* be the first line of your code.
To see the example at work, please review:

http://www.xn--ovg.com/session

tedd

--- copy below and save as session1.php -----
session_start();
?>

"http://www.w3.org/TR/html4/strict.dtd">

Stuff by tedd

Anand's session stuff (page 1)

PHPSESSID =

Click the next page and see numbr of visits during this visit.

Next page

--- copy below and save as session2.php -----

session_start();
($_SESSION['count']) ? $_SESSION['count']++ : $_SESSION['count'] = 1;
?>

"http://www.w3.org/TR/html4/strict.dtd">

Stuff by tedd

Anand's session stuff (page 2)

PHPSESSID =

You have been here times in this session.

Test the SMTP Service on WIndows

Test the SMTP Service

To test the SMTP service, follow these steps:
1. On a computer running Windows Server 2003, type Telnet at a command prompt, and then press ENTER.
2. At the telnet prompt, type set LocalEcho, press ENTER, and then type open 25, and then press ENTER.

The output resembles the following:

220 computername.microsoft.com ESMTP Server (Microsoft Exchange Internet Mail Service 5.5.2651.58) ready

3. Type helo me, and then press ENTER.

The output resembles the following:

250 OK

4. Type mail from:email@domain.com, and then press ENTER.

The output resembles the following:

250 OK - mail from

5. Type rcpt to:youremail@yourdomain.com, and then press ENTER.

The output resembles the following:

250 OK - Recipient

6. Type Data, and then press ENTER.

The output resembles the following:

354 Send data. End with CRLF.CRLF

7. Type Subject:This is a test, and then press ENTER two times.
8. Type Testing, and then press ENTER.
9. Press ENTER, type a period (.), and then press ENTER.

The output resembles the following:

250 OK

10. Type quit, and then press ENTER.
The output resembles the following:

221 Closing Port / Mail queued for delivery

Disallow download .MDB file

Disallow download of certain file type

If what you want to do is prevent a download of your .mdb file,
just create an App_Data directory and place the Access database in it.

Files stored in the App_Data folder are not returned in response to direct HTTP requests,
which makes the App_Data folder the recommended location for data stored with your
application, including .mdf (SQL Server Express Edition), .mdb (Microsoft Access), or XML files.

Command Line Examples linux

Linux command line reference for common operations.

This is a linux command line reference for common operations.
Examples marked with • are valid/safe to paste without modification into a terminal, so
you may want to keep a terminal window open while reading this so you can cut & paste.
All these commands have been tested both on Fedora and Ubuntu.

Command Description
• apropos whatis Show commands pertinent to string. See also threadsafe
• man -t man | ps2pdf - > man.pdf make a pdf of a manual page
which command Show full path name of command
time command See how long a command takes
• time cat Start stopwatch. Ctrl-d to stop. See also sw
• nice info Run a low priority command (The "info" reader in this case)
• renice 19 -p $$ Make shell (script) low priority. Use for non interactive tasks
dir navigation
• cd - Go to previous directory
• cd Go to $HOME directory
(cd dir && command) Go to dir, execute command and return to current dir
• pushd . Put current dir on stack so you can popd back to it
file searching
• alias l='ls -l --color=auto' quick dir listing
• ls -lrt List files by date. See also newest and find_mm_yyyy
• ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
find -name '*.[ch]' | xargs grep -E 'expr' Search 'expr' in this dir and below. See also findrepo
find -type f -print0 | xargs -r0 grep -F 'example' Search all regular files for 'example' in this dir and below
find -maxdepth 1 -type f | xargs grep -F 'example' Search all regular files for 'example' in this dir
find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
• find -type f ! -perm -444 Find files not readable by all (useful for web site)
• find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
• locate -r 'file[^/]*\.txt' Search cached index for names. This re is like glob *file*.txt
• look reference Quickly search (sorted) dictionary for prefix
• grep --color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
gpg -c file Encrypt file
gpg file.gpg Decrypt file
tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
tar -c dir/ | gzip | gpg -c | ssh user@remote 'dd of=dir.tar.gz.gpg' Make encrypted archive of dir/ on remote machine
find dir/ -name '*.txt' | tar -c --files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
find dir/ -name '*.txt' | xargs cp -a --target-directory=dir_txt/ --parents Make copy of subset of dir/ and below
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
( tar -c /dir/to/copy ) | ssh -C user@remote 'cd /where/to/ && tar -x -p' Copy (with permissions) copy/ dir to remote:/where/to/ dir
dd bs=1M if=/dev/sda | gzip | ssh user@remote 'dd of=sda.gz' Backup harddisk to remote machine
rsync (Use the --dry-run option for testing)
rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
rsync --bwlimit=1000 fromfile tofile Locally copy with rate limit. It's like nice for I/O
rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html' Mirror web site (using compression and encryption)
rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
• ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER's home directory on $HOST
ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
wget (multi purpose download tool)
• (cd cli && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/ Download a set of files to the current directory
wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
• wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head Process output directly
echo 'wget url' | at 01:00 Download url at 1AM to current dir
wget --limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
wget -nv --spider --force-html -i bookmarks.html Check links in a file
wget --mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
ethtool eth0 Show status of ethernet interface eth0
ethtool --change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
iwconfig eth1 Show status of wireless interface eth1
iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
• iwlist scan List wireless networks in range
• ip link show List network interfaces
ip link set dev eth0 name wan Rename interface eth0 to wan
ip link set dev eth0 up Bring interface eth0 up (or down)
• ip addr show List addresses for interfaces
ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
• ip route show List routing table
ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
• tc qdisc add dev lo root handle 1:0 netem delay 20msec Add 20ms latency to loopback device (for testing)
• tc qdisc del dev lo root Remove latency added above
• host pixelbeat.org Lookup DNS ip address for name or vice versa
• hostname -i Lookup local ip address (equivalent to host `hostname`)
• whois pixelbeat.org Lookup whois info for hostname or ip address
• netstat -tupl List internet services on a system
• netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
• smbtree Find windows machines. See also findsmb
nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
smbclient -L windows_box List shares on windows machine or samba server
mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
echo 'message' | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout, so if you want to edit files, append newfile)
sed 's/string1/string2/g' Replace string1 with string2
sed 's/$.*$1/\12/g' Modify anystring1 to anystring2
sed '/ *#/d; /^ *$/d' Remove comments and blank lines
sed ':a; /\\$/N; s/\\\n//; ta' Concatenate lines with trailing \
sed 's/[ \t]*$//' Remove trailing spaces from lines
sed 's/$[\\`\\"$\\\\]$/\\\1/g' Escape shell metacharacters active within double quotes
• seq 10 | sed "s/^/ /; s/ *$.\{7,\}$/\1/" Right align numbers
sed -n '1000p;1000q' Print 1000th line
sed -n '10,20p;20q' Print lines 10 to 20
sed -n 's/.*(title)$.*$<\/title>.*/\1/ip;T;q' Extract title from HTML web page
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses
• echo 'Test' | tr '[:lower:]' '[:upper:]' Case conversion
• tr -dc '[:print:]' < /dev/urandom Filter non printable characters
• history | wc -l Count lines
set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)
sort file1 file2 | uniq Union of unsorted files
sort file1 file2 | uniq -d Intersection of unsorted files
sort file1 file1 file2 | uniq -u Difference of unsorted files
sort file1 file2 | uniq -u Symmetric Difference of unsorted files
join -a1 -a2 file1 file2 Union of sorted files
join file1 file2 Intersection of sorted files
join -v2 file1 file2 Difference of sorted files
join -v1 -v2 file1 file2 Symmetric Difference of sorted files
math
• echo '(1 + sqrt(5))/2' | bc -l Quick math (Calculate φ). See also bc
• echo 'pad=20; min=64; (100*10^6)/((pad+min)*8)' | bc More complex (int) e.g. This shows max FastE packet rate
• echo 'pad=20; min=64; print (100E6)/((pad+min)*8)' | python Python handles scientific notation
• echo 'pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)' | gnuplot -persist Plot FastE packet rate vs packet size
• echo 'obase=16; ibase=10; 64206' | bc Base conversion (decimal to hexadecimal)
• echo $((0x2dec)) Base conversion (hex to dec) ((shell arithmetic expansion))
• units -t '100m/9.74s' 'miles/hour' Unit conversion (metric to imperial)
• units -t '500GB' 'GiB' Unit conversion (SI to IEC prefixes)
• units -t '1 googol' Definition lookup
• seq 100 | (tr '\n' +; echo 0) | bc Add a column of numbers. See also add and funcpy
calendar
• cal -3 Display a calendar
• cal 9 1752 Display a calendar for a particular month year
• date -d fri What date is it this friday. See also day
• date --date='25 Dec' +%A What day does xmas fall on, this year
• date --date '1970-01-01 UTC 2147483647 seconds' Convert number of seconds since the epoch to a date
• TZ=':America/Los_Angeles' date What time is it on West coast of US (use tzselect to find TZ)
echo "mail -s 'get the train' P@draigBrady.com < /dev/null" | at 17:45 Email reminder
• echo "DISPLAY=$DISPLAY xmessage cooker" | at "NOW + 30 minutes" Popup reminder
locales
• printf "%'d\n" 1234 Print number with thousands grouping appropriate to locale
• BLOCK_SIZE=\'1 ls -l get ls to do thousands grouping appropriate to locale
• echo "I live in `locale territory`" Extract info from locale database
• LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes
• locale | cut -d= -f1 | xargs locale -kc | less List fields available in locale database
recode (Obsoletes iconv, dos2unix, unix2dos)
• recode -l | less Show available conversions (aliases on each line)
recode windows-1252.. file_to_change.txt Windows "ansi" to local charset (auto does CRLF conversion)
recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset
recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8
recode ../b64 < file.txt > file.b64 Base64 encode
recode /qp.. < file.txt > file.qp Quoted printable decode
recode ..HTML < file.txt > file.html Text to HTML
• recode -lf windows-1252 | grep euro Lookup table of characters
• echo -n 0x80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap
• echo -n 0x20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding
• echo -n 0x20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding
CDs
gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom
mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir
mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only)
cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW
gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom - Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
cdparanoia -B Rip audio tracks from CD to wav files in current dir
cdrecord -v dev=/dev/cdrom -audio *.wav Make audio CD from all wavs in current dir (see also cdrdao)
oggenc --tracknum='track' track.cdda.wav -o 'track.ogg' Make ogg file from wav file
disk space (See also FSlint)
• ls -lSr Show files by size, biggest last
• du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop
• df -h Show free space on mounted filesystems
• df -i Show free inodes on mounted filesystems
• fdisk -l Show disks partitions sizes and types (run as root)
• rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n List all packages by installed size (Bytes) on rpm distros
• dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n List all packages by installed size (KBytes) on deb distros
• dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate
monitoring/debugging
• tail -f /var/log/messages Monitor messages in a log file
• strace -c ls >/dev/null Summarise/profile system calls made by command
• strace -f -e open ls >/dev/null List system calls made by command
• ltrace -f -e getenv ls >/dev/null List library calls made by command
• lsof -p $$ List paths that process id has open
• lsof ~ List processes that have specified path open
• tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me
• ps -e -o pid,args --forest List processes in a hierarchy
• ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' List processes by % cpu usage
• ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem usage. See also ps_mem.py
• ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process
• ps -p 1,2 List info for particular process IDs
• last reboot Show system reboot history
• free -m Show amount of (remaining) RAM (-m displays in MB)
• watch -n1 'cat /proc/interrupts' Watch changeable data continuously
system information (see also sysinfo) ('#' means root access is required)
• uname -a Show kernel version and system architecture
• head -n1 /etc/issue Show name and version of distribution
• cat /proc/partitions Show all partitions registered on the system
• grep MemTotal /proc/meminfo Show RAM total seen by the system
• grep "model name" /proc/cpuinfo Show CPU(s) info
• lspci -tv Show PCI info
• lsusb -tv Show USB info
• mount | column -t List mounted filesystems on the system (and align output)
# dmidecode -q | less Display SMBIOS/DMI information
# smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total
# hdparm -i /dev/sda Show info about disk sda
# hdparm -tT /dev/sda Do a read speed test on disk sda
# badblocks -s /dev/sda Test for unreadable blocks on disk sda
interactive (see also linux keyboard shortcuts)
• readline Line editor used by bash, python, bc, gnuplot, ...
• screen Virtual terminals with detach capability, ...
• mc Powerful file manager that can browse rpm, tar, ftp, ssh, ...
• gnuplot Interactive/scriptable graphing
• links Web browser
miscellaneous
• alias hd='od -Ax -tx1z -v' Handy hexdump. (usage e.g.: • hd /proc/self/cmdline | less)
• alias realpath='readlink -f' Canonicalize path. (usage e.g.: • realpath ~/../$USER)
• set | grep $USER Search current environment
touch -c -t 0304050607 file Set file timestamp (YYMMDDhhmm)

####################################################################################

World of FIND Commands in LINUX

FIND COMMANDS

sudo find / -type f -name *.jpg -exec cp {} . \;

find . -type f -size +10000 -exec ls -al {} \;
find . -atime +1 -type f -exec mv {} TMP \; # mv files older then 1 day to dir TMP
find . -name "-F" -exec rm {} \; # a script error created a file called -F
find . -exec grep -i "vds admin" {} \;
find . \! -name "*.Z" -exec compress -f {} \;
find . -type f \! -name "*.Z" \! -name ".comment" -print | tee -a /tmp/list
find . -name *.ini
find . -exec chmod 775 {} \;
find . -user xuser1 -exec chown -R user2 {} \;
find . -name ebtcom*
find . -name mkbook
find . -exec grep PW0 {} \;
find . -exec grep -i "pw0" {} \;
find . -atime +6
find . -atime +6 -exec ll | more
find . -atime +6 -exec ll | more \;
find . -atime +6 -exec ll \;
find . -atime +6 -exec ls \;
find . -atime +30 -exec ls \;
find . -atime +30 -exec ls \; | wc -l
find . -name auth*
find . -exec grep -i plotme10 {};
find . -exec grep -i plotme10 {} \;
find . -ls -exec grep 'PLOT_FORMAT 22' {} \;
find . -print -exec grep 'PLOT_FORMAT 22' {} \;
find . -print -exec grep 'PLOT_FORMAT' {} \;
find . -print -exec grep 'PLOT_FORMAT' {} \;
find ./machbook -exec chown 184 {} \;
find . \! -name '*.Z' -exec compress {} \;
find . \! -name "*.Z" -exec compress -f {} \;
find /raid/03c/ecn -xdev -type f -print
find /raid/03c/ecn -xdev -path -type f -print
find / -name .ssh* -print | tee -a ssh-stuff
find . -name "*font*"
find . -name hpmcad*
find . -name *fnt*
find . -name hp_mcad* -print
find . -grep Pld {} \;
find . -exec grep Pld {} \;
find . -exec grep Pld {} \;
find . -exec grep PENWIDTH {} \; | more
find . -name config.pro
find . -name config.pro
find /raid -type d ".local_sd_customize" -print
find /raid -type d -name ".local_sd_customize" -print
find /raid -type d -name ".local_sd_customize" -ok cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;
find /raid -type d -name ".local_sd_customize" -exec cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;
find . -name xeroxrelease
find . -exec grep xeroxrelease {} \;
find . -name xeroxrelease
find . -name xeroxrelease* -print 2>/dev/null
find . -name "*release*" 2>/dev/null
find / -name "*xerox*" 2>/dev/null
find . -exec grep -i xeroxrelease {} \;
find . -print -exec grep -i xeroxrelease {} \;
find . -print -exec grep -i xeroxrelease {} \; > xeroxrel.lis
find . -exec grep -i xeroxrel {} \;
find . -print -exec grep -i xeroxrel {} \;
find . -print -exec grep -i xeroxrel {} \; | more
find /raid/03c/inwork -xdev -type f -print >> /raid/04d/user_scripts/prt_list.tmp
find . -exec grep '31.53' {} \;
find . -ls -exec grep "31/.53" {} \; > this.lis
find . -print -exec grep "31/.53" {} \; > this.lis
find . -print -exec grep 31.53 {} \; > this.lis
find . -exec grep -i pen {} /;
find . -exec grep -i pen {} \;
find . -print -exec grep -i pen {} \; | more
find . -exec grep -i pen {} \;
find . -atime +6 -exec ll | more \;
find . -atime +6 -exec ll \;
find . -atime +6 -exec ls \;
find . -atime +30 -exec ls \;
find . -atime +30 -exec ls \; | wc -l
find . \! -name '*.Z' -exec compress -f {} \;
find . -name 'cache*' -depth -exec rm {} \;
find . -name 'cache*' -depth -print | tee -a /tmp/cachefiles
find . -name 'cache[0-9][0-9]*' -depth -print | tee -a /tmp/cachefiles
find . -name 'hp_catfile' 'hp_catlock' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_catfile' -name 'hp_catlock' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_cat*' -depth -print | tee -a /tmp/hp.cats
find . -name 'hp_cat[fl]*' -depth -print | tee -a /tmp/hp.cats
find /raid -name 'hp_cat[fl]*' -depth -print
find . \! -name '*.Z' -exec compress -f {} \;
find . -name '*' -exec compress -f {} \;
find . -xdev -name "wshp1*" -print
find . -xdev -name "wagoneer*" -print
find . -name "xcmd" -depth -print
find /usr/contrib/src -name "xcmd" -depth -print
find /raid -type d -name ".local_sd_customize" -exec ls {} \;
find /raid -type d -name ".local_sd_customize" \
-exec cp /raid/04d/MCAD-apps/I_Custom/SD_custom/site_sd_customize/user_filer_project_dirs {} \;

Qmail Tip sand Tricks 2

Mail to Valid Users Is Bouncing or Disappearing

If you use users/assign as described in Chapter 15, a common mistake is to add a user to the system without updating the users file. Fortunately, this oversight is easily remedied:

# cd /var/qmail/users; make

Delivering Mail on Intermittent Connections

If your qmail system is a hub host for remote systems that connect intermittently by dialup, it is straightforward but messy to deliver the mail while the remote systems are connected.

One approach is to create a flag file in a known directory when a host connects and delete the file when the host disconnects. Then run a script periodically from cron that loops over all of the flag files to push out mail to currently connected hosts.

To flesh out this example, assume there are three dialup hosts called red.example.com, blue.example.com, and green.example.com. Create virtualdomains that give them different virtual domain prefixes:

red.example.com:alias-dial-red
blue.example.com:alias-dial-blue
green.example.com:alias-dial-green

You can put all of the alias-dial mail into one Maildir since the Delivered-To: prefixes keep them separate. To put all the mail for the three hosts into ~alias/dialmail/, create ~alias/.qmail-dial-default containing the line ./dialmail/.

To track the currently connected hosts, put the flag files into ~alias/dialflags and have the dialup connection script create a file with the host's simple name (red, blue, or green) in that directory containing the host's current IP address. Then run this script from cron to push out the mail to whichever hosts are currently connected:

#!/bin/sh
# run this every 15 minutes from cron to push out the mail

cd /var/qmail/alias/dialflags

for hn in *
do
ip=$(cat $hn) # IP address in the flag file

setlock ../$hn.lock \ # lock deliveries to this host
maildirsmtp /var/qmail/alias/dialmail \
alias-dial-$hn- $ip my.example.com 2>&1 |
splogger serial
done

If you also want to push out any waiting mail as soon as a host connects, also put a call to maildirsmtp into the host's connection script. Be sure to use the same lock file to avoid confusion if the cron job happens to run at the same time. If you add another host called purple, you only need to add another line to virtualdomains:

purple.example.com:alias-dial-purple

The remote hosts can use a similar setup to forward their mail to the main host, using a single smarthost entry in virtualdomains.

Code in extra.h to copy everything to log

#define QUEUE_EXTRA "Tlog\0"
#define QUEUE_EXTRALEN 5

Now every message will be copied to the address log, so you can create ~alias/.qmail-log to save the mail:

./logmaildir/

The .qmail file must save the mail but cannot forward it. Why not? Because forwarding mail invokes qmail-queue again, which will redeliver the mail to log, creating a nasty mail loop.

Deleting Stale Mail

cd /home
{
# unread mail over a month old
find /home/*/Maildir/new -type f -mtime +30 -p
# read mail over three months
find /home/*/Maildir/cur -type f -mtime +90 -p
# any mail marked deleted
find */Maildir -type f -name "*:2,*T*" -print
# any mail in Trash/new or cur
find */Maildir/.Trash/??? -type f -print
} | xargs -t rm

Web Sites

There are several excellent sources of qmail information online.

http://cr.yp.to

Dan Bernstein's web site, the official source for qmail and all of his ad-on packages.

http://www.qmail.org

Russ Nelson's qmail resource site, intended to have links to all of the other resources on the Web.

http://qmail.gurus.com

The author's companion site for this book, containing scripts, updates and corrections, links to other resources, and ordering info for more copies.

http://www.lifewithqmail.org

Dave Sill's Life with qmail, an online guide to setting up and using qmail. It offers specific advice about where to install qmail, and where to put all of the files and directories that qmail needs. This is by far the most widely used setup and the one that qmail experts are the most familiar with, so it's the one you should use. The file and directory locations used in this book are consistent with these.

http://www.lifewithqmail.org/ldap/

Henning Brauer's Life with qmail-ldap, a guide to setting up qmail-ldap. Indispensable for qmail-ldap users.

http://www.ezmlm.org

The home page for the ezmlm-idx mailing list manager, with software and documentation.

http://tinydns.org

Russ Nelson's site for Dan Bernstein's djbdns, a DNS package that relates to BIND roughly as qmail relates to sendmail. Not required for qmail, but if you're setting up a DNS server along with your mail server, it's probably the software you want to use.

Qmail Tips and Tricks

Qmail Won't Compile

You have unpacked the qmail sources and typed make, but it won't compile. If you're receiving error messages about errno, you've run into a compatibility problem between qmail and recent versions of the GNU C library. The fix is very simple. See Building with Recent GLIBC and Fixing the errno Problem in Chapter 3.

(This is the number one question on the qmail mailing list, so frequent that there's an autoresponder that mails back the answer to any message that contains the word "errno".)

Why Qmail Is Delivering Mail Very Slowly

If qmail seems to wait about half a minute to do anything when you inject mail, the problem is almost certainly that the lock/trigger file used to communicate between qmail-queue and qmail-send is messed up. That file should be a named pipe:

# ls -l /var/qmail/queue/lock/trigger
prw--w--w- 1 qmails qmail 0 Nov 7 03:02 /var/qmail/queue/lock/trigger

If it's a regular file or anything other than a pipe, you have a problem. Fortunately, it's a problem that's easy to fix:

# svc -td /service/qmail-send # shut qmail down for a minute
# tail -f /service/qmail-send/log/main/current
# # wait until the log says that it's exited
# rm /var/qmail/queue/lock/trigger # remove bogus trigger
# cd wherever you built qmail from source
# make setup check # recreates all the crucial files including trigger
# svc -u /service/qmail-send # restart qmail

This is the second most frequently asked question on the qmail mailing list, and tends to get aggrieved responses pointing out that the answer is in the archives about a hundred times. So don't ask it, because now you know the answer.

Daemons Won't Start, or They Start and Crash Every Few Seconds

Starting a daemon under svscan and supervise is simple in concept, although the details can bite you. The super-daemon is started at system boot time by running /command/svscanboot. It runs svscan to control daemons and the useful but obscure readproctitle, which takes any error messages from svscan and puts them into its command area so that ps will show it.[1]

[1] This odd way of displaying error messages is intended to work even in the presence of serious configuration screwups like disks that should be mounted but aren't and directories that are supposed to be writable but aren't.

Every five seconds svscan looks at all of the subdirectories of /service and starts up a supervise process on any that don't have one running. In the usual case that the subdirectory in turn has a subdirectory called log, it starts a second supervise process in the subdirectory and pipes the output from the first process to the second.

When supervise starts up a daemon, it runs the file run in the daemon's directory. That file has to be a runnable program that either is or, more commonly, exec's the daemon itself. That means that run has to have its execute bits set and, if it's a shell script, start with #!/bin/sh so that it's runnable. If either of those isn't the case, there is a failed attempt to start the daemon every five seconds. A ps l that shows readproctitle should reveal the error messages and give hints about what needs to be fixed.

The run script generally sets up the program environment and then exec's the actual daemon. If you become super-user and type ./run, the daemon should start. If that works, the daemon still doesn't start, and you don't use full program paths in the run file, the problem is most likely that the search path that supervise uses isn't the same as the one you're using. Look at /command/svscanboot to see the search patch that it uses. Most notably, it does not include /var/qmail/bin unless you edit the file yourself to include it.

Nothing Gets Logged

Sometimes the daemon runs but nothing's going into the log files. This generally is due to either file protection problems or an incorrect set of multilog options. The usual way to run multilog is to create a subdirectory called main in which it rotates log files. It's safer to run daemons as a user other than root, so when possible, use qmaill, the qmail log user. A common error is to forget to change the ownership of the log file directory to qmaill (or whatever the log user is). When multilog starts successfully, it creates a current log file in the directory, so if there's no main/current, the most likely problem is directory ownership or protection.

If multilog is running but there's nothing logged, the most likely problems are that the daemon isn't sending anything to log, or that multilog's options are telling it to discard everything. Because the daemon and the logger are connected with a regular Unix pipe, only messages sent to the daemon's standard output go to the logger. In particular, anything sent to standard error shows up in readproctitle, not the log. If, as is usually the case, you want to log the errors a daemon reports, just redirect the error output to the standard output in the run script with the standard shell redirect 2>&1. (That redirect is at the end of just about every run script example in this book.)

If the daemon is a program originally intended to run as a standalone daemon rather than under daemontools, it probably sends its reports to syslog, not to standard output or standard error. In most cases, there is an option to send messages to stdout or stderr.

If you are using multilog options to select what to log, be sure that you're selecting what you think you are. In particular, its pattern language resembles shell wildcards but is in fact considerably weaker because it doesn't move ahead or back up on a failed match. (Patterns do resemble shell wildcards closely enough that they should always be quoted to keep the shell from messing with them.) The pattern must match the whole line, and stars stop matching the moment they see the following character in the pattern. If a pattern is, say, +'+*: status: *', it will match one: status: two, but it will not match one: two: status: three, because the star will stop at the first colon and won't look for the second one. If the pattern didn't have the star at the end, it wouldn't match anything useful because it wouldn't match any lines with anything after the status:. In practice, most log file messages have a pretty simple syntax, and it's not hard to come up with adequate patterns if you keep in mind the limitations of the pattern-matching language. For debugging, start with no patterns to be sure that the stream of messages going into the log files contains what you expect, then add one or two patterns at a time and restart multilog with svc -t and see what's going into main/current each time until it looks right.

Daemons Are Running but Making No Progress

One of the most baffling problems occurs when the daemon seems OK, the logger seems OK, but the daemon's not doing anything. What's wrong? Usually the problem is that the disk to which the log files are written has filled up or is mounted read-only. Because multilog is designed not to lose any log data, if it can't write to the disk, it just waits and retries until it can. This means that the pipe between the daemon and multilog fills up and the daemon stalls waiting to be able to write to the pipe. The solution is to delete some files and fix whatever it was that filled up the disk so it doesn't happen again. If the disk is full of files written by various multilog loggers, adding or adjusting s and n options to set the maximum size and number of log files can help.

Mail Rejected with Stray Newline Reports

The SMTP spec says that the way that each line of text in an SMTP session ends is with a carriage return/line feed pair (0d 0a in hex or \r\n in C.) Some buggy MUAs and MTAs only try to send mail that contains linefeeds with no preceding carriage return. Qmail's SMTP daemon normally rejects such mail with a log message like Stray newline from 10.2.3.4 because there's no way to tell whether the bare linefeed is just missing a carriage return or it's some kind of malformed binary data.

If you're seeing stray newline entries in your logs and you're reasonably sure that they're being sent by MTAs or MUAs that intend them to be handled as an end-of-line, use the fixcrio program from the ucspi-tcp package to placate the SMTP daemon. Modify the run script for qmail-smtpd so that it pipes mail through fixcrio, as shown in Example 18-1:

Example SMTP daemon that forgives stray newlines

1. #!/bin/sh
2. limit datasize 3m
3. exec tcpserver \
4. -u000 -g000 -v -p -R \
5. 0 25 \
6. /usr/local/bin/fixcrio | /var/qmail/bin/qmail-smtpd" 2>&1

Line 6 is the modified one, starting up fixcrio and qmail-smtpd. When fixcrio runs, it passes the input and output of qmail-smtpd through pipes so it can add missing carriage returns in front of newlines as needed. In the longer run, see if you can persuade your correspondents to upgrade their SMTP clients to newer, less buggy versions.

Thursday, February 28, 2008

How Do I Enable remote access to MySQL database server?

Step # 1: Login over ssh if server is outside your IDC

First, login over ssh to remote MySQL database server

Step # 2: Enable networking

Once connected you need edit the mysql configuration file my.cfg using text editor such as vi.

* If you are using Debian Linux file is located at /etc/mysql/my.cnf location
* If you are using Red Hat Linux/Fedora Linux file is located at /etc/my.cnf location
* If you are using FreeBSD you need to create a file /var/db/mysql/my.cnf

# vi /etc/my.cnf
Step # 3: Once file open, locate line that read as

[mysqld]
Make sure line skip-networking is commented (or remove line) and add following line
bind-address=YOUR-SERVER-IP

For example, if your MySQL server IP is 65.55.55.2 then entire block should be look like as follows:
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/English
bind-address = 65.55.55.2
# skip-networking
....
..
....Where,

* bind-address : IP address to bind to.
* skip-networking : Don’t listen for TCP/IP connections at all. All interaction with mysqld must be made via Unix sockets. This option is highly recommended for systems where only local requests are allowed. Since you need to allow remote connection this line should removed from file or put it in comment state.

Step# 4 Save and Close the file

Restart your mysql service to take change in effect:# /etc/init.d/mysql restart
Step # 5 Grant access to remote IP address

# mysql -u root -p mysqlGrant access to new database
If you want to add new database called foo for user bar and remote IP 202.54.10.20 then you need to type following commands at mysql> prompt:mysql> CREATE DATABASE foo;
mysql> GRANT ALL ON foo.* TO bar@'202.54.10.20' IDENTIFIED BY 'PASSWORD';
How Do I Grant access to existing database?

Let us assume that you are always making connection from remote IP called 202.54.10.20 for database called webdb for user webadmin, To grant access to this IP address type the following command At mysql> prompt for existing database:mysql> update db set Host='202.54.10.20' where Db='webdb';
mysql> update user set Host='202.54.10.20' where user='webadmin';
Step # 5: Logout of MySQL

Type exit command to logout mysql:mysql> exit
Step # 6: Test it
From remote system type command:

$ mysql -u webadmin –h 65.55.55.2 –p

Where,

* -u webadmin: webadmin is MySQL username
* -h IP or hostname: 65.55.55.2 is MySQL server IP address or hostname (FQDN)
* -p : Prompt for password

You can also use telnet to connect to port 3306 for testing purpose:
$ telnet 65.55.55.2 3306

SMTP and POP with SSL (ucspi-tls-qmail-howto)

Why You Want To Use UCSPI-TLS

UCSPI-TLS is a protocol for adding "delayed encryption" to Dan Bernstein's Unix Client/Server Program Interface protocol. "Delayed encryption" means a session starts off in plaintext, then a command is issued to turn on encryption, encryption is negotiated, and the session restart. This has become a very common way to handle encryption, because it simplifies client configuration and requires only one TCP port.

These are the goals of UCSPI-TLS:

Simple, UCSPI-compatible use.
Support for both traditional SSL and delayed encryption through STARTTLS or similar.
All SSL/TLS code in one place.
Minimal changes required for each server.
Support for "privilege separation", so that encryption can happen in a low-privilege subprocess.

In particular, I believe the privilege separation feature increases your system's security significantly. It creates a dedicated process to handle each encrypted connection, and this process can change its root directory and switch to a low-privilege user and group. Because of its complexity, OpenSSL has had its share of security bugs. Doing encryption in a low-privilege process ensures that the impact of any security bugs is minimized.

How to install UCSPI-TLS for netqmail-1.05

UCSPI-TLS is currently implemented as patches to existing programs. To install it, first download the programs you want to use, apply the UCSPI-TLS patches, install the programs as you normally would, and finally make the appropriate changes to your configuration files.

Here are more details on doing this with netqmail version 1.05.

Become root, using su or sudo bash. Some of these steps require you to have superuser privileges, and its easier just to get them now than to switch back and forth throughout the process.
Do a base qmail install
Install qmail according to Life with qmail , and make sure everything works with a standard setup.
Install a patched ucspi-ssl+tls
Make sure you have a recent version of OpenSSL installed on your system, along with the files required for development with the library. If you don't already have it installed, this should be available from your OS install CDs, with a name like libssl-dev or openssl-devel. If all else fails, you can download OpenSSL and compile it yourself.
Download ucspi-ssl 0.70, and unpack it. It will create a directory host/superscript.com/net/ucspi-ssl-0.70; cd into this directory. Here's some commands to cut-n-paste:
```
wget http://www.superscript.com/ucspi-ssl/ucspi-ssl-0.70.tar.gz
gunzip -cd ucspi-ssl-0.70.tar.gz |tar xf -
cd host/superscript.com/net/ucspi-ssl-0.70
```

Download ucspi-tls patch to ucspi-ssl (you can also read the ucspitls for ucsp-ssl patch README), and apply the patch:

wget http://www.suspectclass.com/~sgifford/ucspi-tls/files/ucspi-ssl-0.70-ucspitls-0.1.patch
patch -p1 < ucspi-ssl-0.70-ucspitls-0.1.patch

If anything is already using an older copy of sslserver, shut it off now.
Compile and install the patched ucspi-ssl+tls, by running package/compile base then package/install base:
```
package/compile base
package/install base
```
You now have a patched copy of sslserver in /usr/local/bin/sslserver. Congratulations!
Patch qmail
The next step is to patch qmail to add TLS support to its SMTP and POP servers. If you followed Life with qmail religiously, your qmail source code will be in /usr/local/src/netqmail-1.05/netqmail-1.05. cd to that directory, or wherever you have the qmail source:
```
cd /usr/local/src/netqmail-1.05/netqmail-1.05
```

Now download the ucspi-tls patch to netqmail (and read the ucspitls patch for netqmail README), and apply the patch:

wget http://www.suspectclass.com/~sgifford/ucspi-tls/files/netqmail-1.05-ucspitls-0.3.patch
patch -p2 < netqmail-1.05-ucspitls-0.3.patch

If qmail is running, shut it down. On a Life with qmail setup, you do this by running:
```
/var/qmail/bin/qmailctl stop
```
Recompile and re-install qmail by running make setup check:
```
make setup check
```
You now have a copy of qmail-smtpd and qmail-popup with TLS support
Make your certificates
Next you'll need SSL certificates. These certificates contain the encryption keys used to communicate with your servers, and possibly a signature from a trusted authority confirming your server's identity. The first thing you'll need is a place to store them. Let's put them in /var/qmail/ssl:
```
mkdir /var/qmail/ssl
chown root /var/qmail/ssl
chmod 700 /var/qmail/ssl
cd /var/qmail/ssl
```
Create the SSL keys for your system. We'll store the certificate in a file named cert, and the key in key. If you already have certificates you'd like to use, separate out the certificate part (the key part will generally be labeled RSA PRIVATE KEY, and the cert part CERTIFICATE. You can just copy these sections from your certificate file, and paste them into the appropriate files.
You'll have to decide whether you want to use an unsigned certificate or a certificate signed by a Certificate Authority (CA). If you hire a CA to verify your identity and sign your certificate, clients are more likely to accept this certificate without prompting the user. If you use your own unsigned certificate, clients may have to confirm that they trust your certificate.
If you'd like to use a signed certificate, choose a CA, and follow their directions to obtain a signed certificate. Save the certificate in a file called cert, and the key in a file called key. If you just have one file containing both of these, see the beginning of this step for suggestions on how to split it up.
If you'd like to use an unsigned certificate, follow these steps:
1. Make sure openssl is installed on your system.
2. Set a umask that will keep your files protected, by running:
```
umask 077
```
3. Generate a certificate by running:
```
openssl req -new -x509 -keyout key.enc -out cert -days 360
```
  Answer the questions you're asked. Your password is temporary, so it doesn't matter what it is as long as you remember it for a few minutes; you can even write it down. Make sure you use the server's host name, as clients will be configured to connect to it, as the "Common Name" for the certificate. Note that this certificate will expire in 360 days, and you'll have to create a new one before then.
  You'll now have a file called key.enc containing your encrypted key, and a file called cert containing your certificate.
4. Remove the password from your certificate, so the server can start automatically. You can do this by running:
```
openssl rsa -in key.enc -out key
```
  Enter the password you chose above, and you'll have a file called key containing an unencrypted copy of your key.
Create a Diffie-Hellman parameter file. I usually create 1024-bit dhparam files, but I'll admit I don't know exactly what they do. I should probably find out and put it here. You can use the command
```
openssl dhparam -out dhparam 1024
```
to generate a 1024-bit Diffie-Hellman parameter file.
Add a user and group for SSL to drop privileges too; name them both ssl. How to do this depends on your OS and what tools it provides; you can find some examples in the qmail source directory in a file called INSTALL.ids. For example, on Linux you would run:
```
groupadd ssl
useradd -g ssl -d /var/qmail ssl
```
If you can't figure out any OS-specific commands, edit the /etc/group and /etc/passwd files directly, using the file format described in the system manpages for group(5) and passwd(5).

Create a file to set shell variables in /var/qmail/ssl/env. Put these lines in that file:

# Set these three
SSL_USER=ssl
SSL_GROUP=ssl
SSL_DIR=/var/qmail/ssl
# Enable UCSPI-TLS
UCSPITLS=1
# The rest are set based on the above three
SSL_CHROOT="$SSL_DIR"
CERTFILE="$SSL_DIR/cert"
KEYFILE="$SSL_DIR/key"
DHFILE="$SSL_DIR/dhparam"
SSL_UID=`id -u "$SSL_USER"`
if [ $? -ne 0 ]; then echo "No such user '$SSL_USER'" >&2; exit; fi
SSL_GID=`id -g "$SSL_GROUP"`
if [ $? -ne 0 ]; then echo "No such group '$SSL_GROUP'" >&2; exit; fi
# Export the variables used by other scripts
export SSL_CHROOT SSL_UID SSL_GID UCSPITLS CERTFILE KEYFILE DHFILE

Now set your umask back to something more usable, like 022:
```
umask 022
```
Set up qmail-smtpd
Edit the qmail-smtpd run file, in /var/qmail/supervise/qmail-smtpd/run. There are three changes required:
- The top of the file has several variable settings. Below these lines, include the SSL environment variable script we created above, using the shell's "dot" command, typed as a single period:
```
. /var/qmail/ssl/env
```
- On the line that contains softlimit, add 10MB (10,000,000) the number after the -m flag. This allows qmail-smtpd to use the extra memory required for SSL. For example, if it's currently 2000000, you would have for that line:
```
exec /usr/local/bin/softlimit -m 12000000 \
```
- On the line that contains tcpserver, change tcpserver to sslserver -e -n, leaving all of the other flags in place. The line will now look something like:
```
/usr/local/bin/sslserver -e -n -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
```
Restart qmail to use the new configuration:
```
/var/qmail/bin/qmailctl restart
```

Now we'll try a few tests to make sure everything works. If any of these steps fails, double-check the steps above, look for errors in the error logs, and use your common sense to see what's going wrong. Look in the logs to see if there are any errors:

tail /var/log/qmail/smtpd/current

You should see something like this:

@40000000431fb29d10e09c2c sslserver: cafile 32655
@40000000431fb29d10e120fc sslserver: ccafile 32655
@40000000431fb29d10e14bf4 sslserver: cadir 32655 /usr/local/ssl/certs
@40000000431fb29d10e176ec sslserver: cert 32655 /var/qmail/ssl/cert
@40000000431fb29d10e1a9b4 sslserver: key 32655 /var/qmail/ssl/key
@40000000431fb29d10e1d4ac sslserver: param 32655 /var/qmail/ssl/dh 512
@40000000431fb29d10e226b4 sslserver: status: 0/20

Make sure the server is now offering TLS. To test this, you can connect to the server on port 25 and issue an "extended HELO" command (EHLO) and see what extensions the server offers. To do this, start at a shell and type in the lines marked below with *:
```
  * telnet localhost 25
   Trying 127.0.0.1...
   Connected to localhost.
   Escape character is '^]'.
   220 host.example.com ESMTP
 * EHLO there
   250-host.example.com
   250-PIPELINING
   250-8BITMIME
   250 STARTTLS
 * QUIT
   221 host.example.com
```
The line that says STARTTLS after you type EHLO means that the server has TLS configured.
Now test with a client to make sure things basically work. First start watching the logfile for errors:
```
tail --follow=name /var/qmail/supervise/qmail-smtpd/log/main/current
```
Unfortunately the client included with sslserver doesn't support TLS. If you have a copy of stunnel version 3 lying around, you can do something like this:
```
stunnel -r localhost:25 -f -c -n smtp -D debug -P none
```
Otherwise, just skip this test and try the test in the next step.
Finally, try turning on TLS in the mail clients you'll be using and sending a test message. Make sure there are no errors from the client or in the logs, and that the messages arrives successfully.
If this step works, you've set everything up correctly. Congratulations!
Set up qmail-pop3d
If you want to run a POP3 server, make sure you've set up qmail-pop3d as described in Life with qmail, then continue to follow these steps. If you're not running a POP3 server, you can skip this entire section.
Edit the qmail-pop3d run file, in /var/qmail/supervise/qmail-pop3d/run. There are three changes required:
- Near the top of the file, between the #!/bin/sh line and the line that begins with exec, include the SSL environment variable script we created above, using the shell's "dot" command, typed as a single period:
  . /var/qmail/ssl/env
- On the line that contains softlimit, add 10MB (10,000,000) the number after the -m flag. This allows qmail-smtpd to use the extra memory required for SSL. For example, if it's currently 2000000, you would have for that line:
  exec /usr/local/bin/softlimit -m 12000000 \
- On the line that contains tcpserver, change tcpserver to sslserver -e -n, leaving all of the other flags in place. The line will now look something like:
  /usr/local/bin/sslserver -e -n -v -R -H -l 0 0 110 /var/qmail/bin/qmail-popup \
Restart qmail to use the new configuration:
```
/var/qmail/bin/qmailctl restart
```
Now we'll try a few tests to make sure everything works. If any of these steps fails, double-check the steps above, look for errors in the error logs, and use your common sense to see what's going wrong. Look in the logs to see if there are any errors:
```
tail /var/log/qmail/pop3d/current
```
You should see something like this:
```
@40000000431fb29d10e09c2c sslserver: cafile 32655
@40000000431fb29d10e120fc sslserver: ccafile 32655
@40000000431fb29d10e14bf4 sslserver: cadir 32655 /usr/local/ssl/certs
@40000000431fb29d10e176ec sslserver: cert 32655 /var/qmail/ssl/cert
@40000000431fb29d10e1a9b4 sslserver: key 32655 /var/qmail/ssl/key
@40000000431fb29d10e1d4ac sslserver: param 32655 /var/qmail/ssl/dh 512
@40000000431fb29d10e226b4 sslserver: status: 0/20
```
Make sure the server is now offering TLS. To test this, you can connect to the server on port 110, then ask for a list of its capabilities. To do this, start at a shell, and type in the lines marked below with *:
```
  * telnet localhost 110
   Trying 127.0.0.1...
   Connected to localhost.
   Escape character is '^]'.
   +OK
 * CAPA
   +OK capability list follows
   STLS
   .
 * QUIT
   +OK
   Connection closed by foreign host.
```
The STLS line after you type CAPA indicates that the server supports TLS.
Now test with a client to make sure things basically work. First start watching the logfile for errors:
```
tail --follow=name /var/qmail/supervise/qmail-pop3d/log/main/current
```
Unfortunately the client included with sslserver doesn't support TLS. If you have a copy of stunnel version 3 lying around, you can do something like this:
```
stunnel -r localhost:110 -f -c -n pop3 -D debug -P none
```
Otherwise, just skip this test and try the test in the next step.
Finally, try turning on TLS in the mail clients you'll be using and receiving some test messages. Make sure there are no errors from the client or in the logs, and that the messages arrives successfully.
If this step works, you've set everything up correctly. Congratulations!

Optional Step: If desired, set up SSL servers

If you'd like to support SSL (not delayed encryption through TLS), you can still use the modified sslserver, and get the security advantages of chroot and privilege separation.

These steps will help you create an SSL service from a TLS service. The instructions are for qmail-smtpd; they will work for qmail-pop3d if you simply replace smtpd with pop3d everywhere.

Change directory to /var/qmail/supervise:
```
cd /var/qmail/supervise
```
Create a service directory and log directory:
```
mkdir -p qmail-smtpd-ssl/log
```
Copy the run file from the original service, and make it executable:
```
cp qmail-smtpd/run qmail-smtpd-ssl/
chmod 755 qmail-smtpd-ssl/run
```
Edit the run file (qmail-smtpd-ssl/run) in the following ways:
- On the line after . /var/qmail/ssl/env, add unset UCSPITLS
- On the line that contains sslserver line, remove the -n flag.
- On the next line, which will contain something like 0 smtp, change smtp to smtps; that tells sslserver to listen on the appropriate port for the SSL version of this service.

Set up a logging directory for this new service:

mkdir /var/log/qmail/smtpd-ssl
chown qmaill /var/log/qmail/smtpd-ssl

Set up the logging program for this new service, by creating a file in qmail-smtpd-ssl/log/run with these contents:
```
#!/bin/sh
exec /usr/local/bin/setuidgid qmaill /usr/local/bin/multilog t \
      /var/log/qmail/smtpd-ssl
```
Make sure the script is executable:
```
chmod 755 qmail-smtpd-ssl/log/run
```
Link the new service into the /service directory, to have it start automatically on boot:
```
ln -s /var/qmail/supervise/qmail-smtpd-ssl /service
```

Add the following to qmailctl's "start" section:

if svok /service/qmail-smtpd-ssl ; then
 svc -u /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log
else
 echo qmail-smtpd-ssl supervise not running
fi

Add the following to qmailctl's "stop" section:

echo "  qmail-smtpd-ssl"
svc -d /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log

Add the following to qmailctl's "stat" section:

svstat /service/qmail-smtpd-ssl
svstat /service/qmail-smtpd-ssl/log

Add the following to qmailctl's "pause" section:

echo "Pausing qmail-smtpd-ssl"
svc -p /service/qmail-smtpd-ssl

Add the following to qmailctl's "cont" section:

echo "Continuing qmail-smtpd-ssl"
svc -c /service/qmail-smtpd-ssl

Add the following to qmailctl's "restart" section:

echo "* Restarting qmail-smtpd-ssl."
svc -t /service/qmail-smtpd-ssl /service/qmail-smtpd-ssl/log

To test, use an SSL-enabled mail client, or use stunnel version 3:
```
stunnel -r localhost:smtps -f -c -D debug -P none
```

Optional Step: If desired, set up other servers

If you have any other TLS or SSL servers you'd like to set up, such as an IMAP server, you should be able to use the above steps as a template to get you started.

Required Step: Sleep better at night

Knowing your TLS- and SSL-protected mail server is secure.

Thanks

Rbldns How to

What is this document?

I wrote this HOWTO so that others can have one place to stop and see how to setup their own private RBL list. The information is out there, but scattered all over the place.

The color convention I use is to have the headlines in gray and the code / shell scripts in yellow

What is rbldns?

Rbldns is a suite of programs in conjunction with dnscache to provide RBL service. In a nutshell IPs put into the rbldns data file show up as 127.0.0.1 responses to a specially crafted DNS query.

The IP address go into the file in the normal readable way, ie 192.168.0.1, and you query for them by reversing the address, like 1.0.168.192.example.com

How do I use rbldns?

Normally you don't use it directly, you have a program query an RBL server and based on the result of the query it either runs or doesn't run the program following it.

For example with Qmail you'll change your qmail-smtpd/run file from
tcpserver 0 smtp /var/qmail/bin/qmail-smtpd
to
tcpserver 0 smtp /usr/local/bin/rblsmtpd -a whitelist.example.com -r blacklist.example.com -r relays.ordb.org /var/qmail/bin/qmail-smtpd

According to the rblsmtpd man page, the -r and -a switches mean:

Switch	If successful, do this
-r	Quit
-a	Don't do other lookups, and continue on

In this case the steps are:
1. Do a lookup against whitelist.example.com, if sucessful then skip the other lookups and go on to accepting the smtp request
2. Do a lookup against blacklist.example.com, if that's successful then quit out printing a "RBL denied" message to the client.
3. Same as #2 but against relays.ordb.org

How do I make my own RBL?

You should know how to setup Dnscache before trying to do this. The steps are pretty easy, but its helpful if you know your way around

Setting up the rbldns is pretty easy, just follow the directions on the RBLdns-conf page:

 rbldns-conf rbldns dnslog /etc/rbldns/black 127.0.0.2 rbl.example.com
rbldns-conf rbldns dnslog /etc/rbldns/white 127.0.0.3 whitelist.example.com

This says to setup a RBL on 127.0.0.2 that answers queries for *.rbl.example.com, and and RBL on 127.0.0.3 that answers for *.whitelist.example.com

Note: the 'rbl' and 'whitelist' names are arbitrary. RBLdns doesn't know if this is a 'good' or a 'bad' list of addresses -- it just knows that if someone asks for 1.0.168.192.bad.example.com on IP 127.0.0.2 it should answer with a 127.0.0.1 if it has it, or nothing if it doesn't.

Start these services up like all of DJB's other daemons:

 ln -s /etc/rbldns/black /service
ln -s /etc/rbldns/white /service

To see if they are running, do a

 lsof -n -i:53

And you should see listeners on 127.0.0.2 and 127.0.0.3

Why would I want to use a local RBL?

Why shouldn't you just relay on publically available RBLs? For a couple of reasons:
1. You know a couple of IPs are good and you don't want to waste bandwith by checking them against an RBL on the internet
2. Same thing for bad ones
3. You know your friend has a DSL line at home and an RBL blocks all home DSL users, you want to continue using the service but want to accept his mail.
4. You want to keep a 2nd source of information about who is sending mail to your server

So part of this is being a good netizen. If you know you're getting a lot of email from a handful of IPs, you should put those in your own whitelist so that you're not continously asking the public RBLs if they are okay

How to resolve *.(whitelist|rbl).example.com -- DNScache way

Now that we have the RBL service ready to go we have to be able to query it. Like we told dnscache to send all queries for *.example.com to the local tinydns IP address, we're going to do that for the RBL service.

This is the part that took me a while to think about: how do I serve both *.example.com and *.rbl.example.com? Well the same way you've already done it: through a dnscache setting.

Add two entries to your dnscache "forwarding" information:

 cd /service/dnscache/root/servers
echo 127.0.0.2 > rbl.example.com
echo 127.0.0.3 > whitelist.example.com
svc -h /service/dnscache

This tells dnscache that when it gets a query for *.whitelist.example.com to send it off to the server on 127.0.0.3, *.rbl.example.com queries should go to 127.0.0.2. If you're setup TinyDns this should be familiar

How to resolve *.(whitelist|rbl).example.com -- TinyDns way

This might be a little cleaner if you're already serving up *.example.com results. You put in your tinydns file pointers to the (rbl|whitelist) lists:

 cd /service/tinydns/root/
echo "&rbl.example.com:127.0.0.2:a" >> data
echo "&whitelist.example.com:127.0.0.3:a" >> data
make
svc -h /service/tinydns

So now when your tinydns server is queried for 1.0.168.192.rbl.example.com it will look at its file and say "That's handled by 127.0.0.2 and I'll ask it"

How do I add entries to my RBL?

Scenerio: a spammer is continously sending us email from 192.168.5.100 while our friend is sending from 192.168.6.56. We would like to avoid using public RBL servers for these two addresses.

Step 1: add the spammer to your spam list:

 cd /service/rbl/black/root
echo 192.168.5.100 >> data
make

Step 2: Add your friend to your good list:

 cd /service/rbl/white/root
echo 192.168.6.56 >> data
make

Now query to see if they are in there:

 dnsip 100.5.168.192.rbl.example.com
dnsip 100.5.168.192.whitelist.example.com
dnsip 56.6.168.192.rbl.example.com
dnsip 56.6.168.192.whitelist.example.com

Notice I queried both sets of databases to make sure we didn't screw up who is a spamer versus who is our friend.

If you don't get a 127.0.0.1 result from a query when you think you should, HUP the appropriate daemon:

 svc -h /service/rbl/black
svc -h /service/rbl/white

How do I now use an RBL service?

Now that you have the local RBLdns server(s) working through simple queries, modify your other scripts to use them. With qmail you do that like this:

exec /usr/local/bin/tcpserver -R -v -p \
  -x /var/qmail/control/tcp.smtp.cdb \
  -u $QMAILUID -g $QMAILGID \
  mail.example.com smtp \
  /usr/local/bin/rblsmtpd \
  -a whitelist.example.com \
  -r rbl.example.com \
  -r relays.ordb.org \
  -r bl.spamcop.net \
  /var/qmail/bin/qmail-smtpd 2>&1

This is what your implementation will probably look like: an initial test against your goodlist, if that passes the continue on to the qmail-smtpd part. If that fails, then test against your local bad list. If that is sucessful then quit out. If not, continue on to your 2nd and 3rd RBLs.

How do I test my RBL?

You test an RBL by sending it a query. If it is in the database then it should return with a 127.0.0.x (1 or 2) answer. If it isn't in there, then nothing comes back.

The trick is to remember to reverse your IP address, so that 192.168.0.1 becomes 1.0.168.192.myrbldomain.com

Here's a simple program to check names fed on stdin:

 #!/usr/bin/perl

use Net::DNS;
use strict;
my ($res, @rbls);

@rbls = qw (rbl.example.com whitelist.example.com);

$res  = Net::DNS::Resolver->new;
while (<>) {
 my ($rev, $ip);
 chomp;
 $ip = $_;
 $rev = (join ".", reverse (split /\./, $ip) );  # ie 4.3.2.1.
 foreach my $rbl (@rbls) {
  my $query = $res->search("$rev.$rbl");
  next unless ($query);
  my $hit;
  foreach my $rr ($query->answer) {
   next unless $rr->type eq "A";
   $hit = $rr->address;
   last;
  }
  next unless $hit;
  printf "%-20s %-20s $hit\n", $ip, $rbl;
 }
}

How do I contact you?

If you have any questions or comments, or just want to say thanks, drop me a line at anand.shah@coolaquarius.com

Thursday, May 22, 2008

Wednesday, March 26, 2008

Anand's session stuff (page 1)

PHPSESSID =

Click the next page and see numbr of visits during this visit.

Anand's session stuff (page 2)

PHPSESSID =

You have been here times in this session.

Test the SMTP Service

Disallow download of certain file type

Linux command line reference for common operations.

FIND COMMANDS

Mail to Valid Users Is Bouncing or Disappearing

Delivering Mail on Intermittent Connections

Code in extra.h to copy everything to log

Deleting Stale Mail

Web Sites

Qmail Won't Compile

Why Qmail Is Delivering Mail Very Slowly

Daemons Won't Start, or They Start and Crash Every Few Seconds

Nothing Gets Logged

Daemons Are Running but Making No Progress

Mail Rejected with Stray Newline Reports

Example SMTP daemon that forgives stray newlines

Thursday, February 28, 2008

Why You Want To Use UCSPI-TLS

How to install UCSPI-TLS for netqmail-1.05

Do a base qmail install

Install a patched ucspi-ssl+tls

Patch qmail

Make your certificates

Set up qmail-smtpd

Set up qmail-pop3d

Optional Step: If desired, set up SSL servers

Optional Step: If desired, set up other servers

Required Step: Sleep better at night

Thanks

What is this document?

What is rbldns?

How do I use rbldns?

How do I make my own RBL?

Why would I want to use a local RBL?

How to resolve *.(whitelist|rbl).example.com -- DNScache way

How to resolve *.(whitelist|rbl).example.com -- TinyDns way

How do I add entries to my RBL?

How do I now use an RBL service?

How do I test my RBL?

How do I contact you?

Blog Archive

About Me

Who will Win IPL season 2