Solution

For exactly this reason i wrote a helper script, in this case it is the list-rules script, which takes over the list of chains and entries that iptables returns and manipulate it to be numbered. This makes it verry easy and handy to delete a rule by rule-number.

The script: list-rules

Let’s have a look to the list-rules script (sourcode is attached below). The chain parameter is always accepted by the script. Assume that you want to show just the INPUT chain, you should give this parameter to list-rules.

list-rules INPUT

gives you a list like that:

Chain INPUT (policy ACCEPT)
Num target     prot opt source               destination         
1 fail2ban-ssh  tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 22 
...
10 DROP       udp  --  229.229.1.2          0.0.0.0/0  
11 DROP       tcp  --  229.229.1.2          0.0.0.0/0  

to delete an entry you could enter the whole iptables rule:

iptables --delete INPUT -d 229.229.1.2/32 -p udp -j DROP
iptables --delete INPUT -d 229.229.1.2/32 -p tcp -j DROP

or with the rule-number:

iptables --delete INPUT 10
iptables --delete INPUT 10

If you would list the rules again, you’ll see the rules are no more listed as they where deleted:

Chain INPUT (policy ACCEPT)
Num target     prot opt source               destination         
1 fail2ban-ssh  tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 22 

Extended chains

There are more chains supported then the INPUT/OUTPUT chains. Have another look to the script and find the $LISTCHAINS variable. You are able to list the extended chains there and use them as parameter to the list-rules script. Always keep in mind, that the default parameter is ALL, which is an keyword to show a list of all chains and rules.

Sourcode: list-rules

#!/usr/bin/env sh
# ######################################################
# Category: Firewall
# Related: iptables
# Script-Name:list-rules 
# ######################################################
# This script is intended to users which are working on 
# the console and configuring iptables chains and rules. 
# It lists all chains and prefix the rule number, which 
# makes it more suitable for deletions.
# ######################################################
# Author: Markus Niewerth 
# Date: Thu, 23 Aug 2012 22:16:42 +0200 
# Copyright: 2011-2012 by Markus Niewerth
# License: GPLv3
# ######################################################
# Default chain is ALL which is a macro for all chains; 
# use the $LISTCHAINS param to configure chains.
# FIXME: export $LISTCHAINS to 
#   /etc/default/list-rules-default.conf
# ######################################################

chain=${1:-ALL}
LISTCHAINS="fail2ban-ssh fail2ban-postfix fail2ban-courierauth \
		fail2ban-ssh-ddos fail2ban-couriersmtp"

if [ $chain = ALL ]; then
	# basic chains
	INPUT=`$0 INPUT`
	OUTPUT=`$0 OUTPUT`
	FORWARD=`$0 FORWARD`
	# extended chains $LISTCHAINS
	for LSC in $LISTCHAINS; do
		_LISTCHAINS="$_LISTCHAINS `$0 $LSC` \n"
	done;

	echo -e "$INPUT \n$OUTPUT \n$FORWARD \n$_LISTCHAINS"
	exit 0;
fi;

OIFS=$IFS
IFS=$'\n'

# first lines are trash
i=-2

for line in `/sbin/iptables --list $chain --numeric`; do 
	((i++))
    if [ $i = 0 ]; then
    	echo -n "Num "
    fi
    
    if [ $i -gt 0 ]; then
    	echo -n "$i "
    fi
    
    echo "$line"
done

# reset to old IFS
IFS=$OIFS

# exit script with the last exit value (just useful for 
#  single chains)
exit $?

Alternatives

Another alternative is to use the iptables-save output, to become the corresponding rules copied or deleted. You should try this out by typing:

iptables-save

Download Script

Size: 985 bytes
Info: bzip2 compressed data, block size = 900k
md5sum: 293e45e091f4fc54a5454fd2fbc39a7d
File: list-rules.tar.bz2

Anyways let’s have a look what the Makefile is able to do:

- clean (cleans everything but archive)
- create-structure (create archive structure)
- backup-all (Performs every of target below)
  - backup-svn
  - backup-ldap
  - backup-mysql
  - backup-www (Performs everything of targets below)
    - www-structure 
    - www-config 
    - www-data 
    - www-archive
  - backup-ldap
- archive (Create the main archive)
- clean-svn (Clean backup files)
- clean-ldap (Clean backup files)
- clean-mysql (Clean backup files)
- clean-www (Clean backup files)
- clean-etc (Clean backup files)
- clean-archive-ldap (Clean archive files)
- clean-archive-mysql (Clean archive files)
- clean-archive-www (Clean archive files)
- clean-archive-etc (Clean archive files)
- disk-usage (Shows disk usage)
- list-archive (Lists and totals archive)
- get-archive (Client sided download target)

While each target is server sided, there is one target for the client side. The target is get-archive and will download the latest created backup archive. While each of the backup targets ldap,mysql,svn,etc are mostly identical on GNU Linux systems, the www backup task is not. So let’s have an eye on the Makefile and the configuration part:

export PATH:=$(CURDIR)/bin:$(PATH)

ifndef TARGETS
TARGETS=clean create-structure backup-all archive
endif

ifndef DATETIME
# an identifier - i assumed to create backups once a day
DATETIME=$(shell date +%Y%m%d)
endif

# just store it, mabye we need the macro later...
OLDCURDIR=$(CURDIR)

# configuration parameters
BACKUPDIR=/backups
SVNADMIN=/usr/bin/svnadmin
SVNREPODIR=/srv/vcs/svn
BACKUPNAME=example.com
REMOTE_SERVER=root@example.com
REMOTE_BACKUP_DIR=/var/backups/example.com
DOMAINS=example.com www2.example.com
WWWDATADOMAINS=example.com
VHOSTSDIR=/var/www
VERBOSE=v

Macro Description

The macro TARGETS and DATETIME are configurable before the make command.

TARGETS: clean create-structure backup-all archive
DATETIME: $(shell date +%Y%m%d)

For instance:
$ TARGETS="clean backup-all" DATETIME=20121010 make

Configuration Macros

BACKUPDIR: /backups
This describes the working directory on the remote server where backups are stored.

SVNADMIN: /usr/bin/svnadmin
This defines the svnadmin command.

SVNREPODIR: /srv/vcs/svn
Basic repository path. This is the path where svn stores all repositories.

BACKUPNAME: example.com
A static identifier name to be used as the backup archive name, prefixed by DATETIME.

REMOTE_SERVER: root@example.com
The user/server on the server side should be root, while the user that downloads (scp) the archive could also be someone else.

REMOTE_BACKUP_DIR: /var/backups/example.com
If a remote backup is stored on the local client machine, use this folder for downloads (scp).

DOMAINS: example.com www2.example.com
This domains are needed to store configurations for.

WWWDATADOMAINS: example.com
While this domains are used to backup just the www data. This could differ and often the webcontents are already backuped on local machines.

VHOSTSDIR: /var/www
This is the path where all vhosts are stored. The DOMAINS macro describes the folders inside this path.

VERBOSE: v
Should we verbose output everything. This should be enabled.

The Makefile Targets

Now let’s have an eye on the targets i have defined for backup purposes. The targets backup-all is having subtargets which also could called manually. In general it is always possible to call targets manual and recreate backups manually on the same day (see DATETIME macro).

.PHONY: clean

all: $(TARGETS) 

create-structure:
	mkdir -p$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/{svn,ldap,mysql,www,etc}

backup-all: backup-svn backup-ldap backup-mysql backup-www backup-etc 
	
backup-svn: 
	if [ ! -d $(SVNREPODIR) ]; then \
		exit 10; \
	fi; \
    for repo in $(shell ls -1 $(SVNREPODIR)); do \
    	if [ ! -f $(SVNREPODIR)/$$repo/format ]; then \
            continue; \
        fi; \
        $(SVNADMIN) dump $(SVNREPODIR)/$$repo | gzip -c -9 > $(BACKUPDIR)/$(DATETIME)/svn/$$repo.gz; \
	done;
	
backup-ldap: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/ldap/var/lib/ldap ]; then \
	 	rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap/* ; \
	fi; \
	mkdir -p$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap/{var/lib,usr/share/phpldapadmin}; \
	/etc/init.d/ldap stop;  \
	sleep 2; \
	cp -pR$(VERBOSE) /var/lib/ldap  $(BACKUPDIR)/$(DATETIME)/ldap/var/lib/ ; \
	/etc/init.d/ldap start; \
	cp -pR$(VERBOSE) /usr/share/phpldapadmin  $(BACKUPDIR)/$(DATETIME)/ldap/usr/share/ ; \
	tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap.tar.bz2  $(BACKUPDIR)/$(DATETIME)/ldap; \
	[ -d $(BACKUPDIR)/$(DATETIME)/ldap ] && rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap;

backup-mysql:
	if [ -d $(BACKUPDIR)/$(DATETIME)/mysql/var/lib/mysql ]; then \
	 	rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql/* ; \
	fi; \
	mkdir -p$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql/{var/lib,usr/share}; \
	/etc/init.d/mysql stop;  \
	cp -pR$(VERBOSE) /var/lib/mysql  $(BACKUPDIR)/$(DATETIME)/mysql/var/lib/ ; \
	/etc/init.d/mysql start;  \
	cp -pR$(VERBOSE) /usr/share/mysql  $(BACKUPDIR)/$(DATETIME)/mysql/usr/share/ ;
	tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql.tar.bz2  $(BACKUPDIR)/$(DATETIME)/mysql; \
	[ -d $(BACKUPDIR)/$(DATETIME)/mysql ] && rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql;
	
backup-www: www-structure www-config www-data www-archive

www-structure:
	for d in $(DOMAINS); do \
		mkdir -p$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www/$$d; \
    done;

www-config:
	for d in $(DOMAINS); do \
		cp -pR$(VERBOSE) $(VHOSTSDIR)/$$d/{conf,pd} $(BACKUPDIR)/$(DATETIME)/www/$$d/ ; \
	done; 

www-data:
	for d in $(WWWDATADOMAINS); do \
		tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www/$$d/httpdocs.tar.bz2 $(VHOSTSDIR)/$$d/httpdocs; \
		if [ -d $(VHOSTSDIR)/$$d/usr ]; then \
			tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www/$$d/usr.tar.bz2 $(VHOSTSDIR)/$$d/usr; \
		fi; \
	done; 	
	
www-archive:
	tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www.tar.bz2  $(BACKUPDIR)/$(DATETIME)/www; \
	[ -d $(BACKUPDIR)/$(DATETIME)/www ] && rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www;

backup-etc:
	cp -pR$(VERBOSE) /etc/*  $(BACKUPDIR)/$(DATETIME)/etc/ ; \
	tar cjf$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/etc.tar.bz2  $(BACKUPDIR)/$(DATETIME)/etc; \
	[ -d $(BACKUPDIR)/$(DATETIME)/etc ] && rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/etc;

archive:
	if [ -L $(BACKUPDIR)/latest-backup ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/latest-backup; \
	fi; \
	if [ -f $(BACKUPDIR)/$(DATETIME)-$(BACKUPNAME).tar.bz2 ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)-$(BACKUPNAME).tar.bz2; \
	fi; \
	if [ -d $(BACKUPDIR)/$(DATETIME) ]; then \
		tar cjf$(VERBOSE)  $(BACKUPDIR)/$(DATETIME)-$(BACKUPNAME).tar.bz2 $(BACKUPDIR)/$(DATETIME); \
		ln -s$(VERBOSE) $(BACKUPDIR)/$(DATETIME)-$(BACKUPNAME).tar.bz2 $(BACKUPDIR)/latest-backup; \
	fi;

clean-svn: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/svn ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/svn/*.gz; \ 
	fi;

clean-ldap: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/ldap ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap/*.*; \
	fi;
	
clean-mysql: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/mysql ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql/*.*; \
	fi;
	
clean-www: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/www ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www/*.*; \
	fi;	
	
clean-etc: 
	if [ -d $(BACKUPDIR)/$(DATETIME)/etc ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/etc/*.*; \
	fi;

clean-archive-ldap: 
	if [ -f $(BACKUPDIR)/$(DATETIME)/ldap.tar.bz2 ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/ldap.tar.bz2; \
	fi;
	
clean-archive-mysql: 
	if [ -f $(BACKUPDIR)/$(DATETIME)/mysql.tar.bz2 ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/mysql.tar.bz2; \
	fi;
	
clean-archive-www: 
	if [ -f $(BACKUPDIR)/$(DATETIME)/www.tar.bz2 ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/www.tar.bz2; \
	fi;
	
clean-archive-etc: 
	if [ -f $(BACKUPDIR)/$(DATETIME)/etc.tar.bz2 ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME)/etc.tar.bz2; \
	fi;
		
clean-latest:
	if [ -f `readlink $(BACKUPDIR)/latest-backup` ]; then \
		rm -r$(VERBOSE) `readlink $(BACKUPDIR)/latest-backup`; \
	fi; \
	if [ -L $(BACKUPDIR)/latest-backup ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/latest-backup; \
	fi;
	
clean:
	if [ -d $(BACKUPDIR)/$(DATETIME) ]; then \
		rm -r$(VERBOSE) $(BACKUPDIR)/$(DATETIME); \
	fi;

disk-usage:
	du -sh $(BACKUPDIR)/$(DATETIME)/{*.bz2,svn/,../*-$(BACKUPNAME).tar.bz2}
	
list-archive:
	tar tjfv --totals `readlink $(BACKUPDIR)/latest-backup` --totals
	
get-archive:
	if [ ! -d $(REMOTE_BACKUP_DIR) ]; then \
		mkdir $(REMOTE_BACKUP_DIR); \
	fi; \
	scp $(REMOTE_SERVER):$(BACKUPDIR)/latest-backup $(REMOTE_BACKUP_DIR)/$(DATETIME)-$(BACKUPNAME).tar.gz;

The above code needs some explanations. When calling make the macro TARGET will be used to define the tasks. If starting a backup keep in mind that the backup directory is cleaned completely. You could modify this for instance with rsync. If you would use rsync for synchronizing instead of doublicating the files, it would be possible to remove the clean target from the TARGET macro.

The create-structure target is intended to create or recreate the folders where files are backuped to and later archived. The backup targets mostly working in the same way, they stop a running service (apache, mysql or ldap) and copy over library and share pathes. In the case of the ldap task it backups the phpldapadmin too, if you’re not using the ldapadmin you have to remove the line.

The apache server backup task works more complicated and is executed in serveral steps. First of all it creates the backup structure for each vhost. The www-config target is just in case you have Plesk runnning or a configuration folder for each vhost. In the next step the htdocs data is backuped (see macro description WWWDATADOMAINS), the routines may differ if you are using other configurations, rewrite this from scratch if you want.

Each backup task is using tar and bzip2 in the end to compress the backup data. The files are called ldap.tar.bz2, mysql.tar.bz2, svn.tar.bz2, www.tar.bz2, etc.tar.bz2 and are later again archived to become a single file.

Download Makefile

The whole file could be downloaded here. Extract the Makefile to a folder of your choice. It is not necesarry to use the current path (/usr/share). To start the whole backup perform just make.

Do the same on the client machine. Extract the Makefile and download the archive with:

$ make get-archive

If you have any suggestions or quetsions feel free to ask.

Additional Documents

* GNU Make: gnu/make.html
* GNU Automake: gnu/automake.html

– Last updated: Sat, 22 Aug 2012 16:58:46 +0200 (mniewerth)

Many thanks to Peter Palfrader for providing the LDAP schema and especially to David Shaw for providing invaluable help and adding LDAP basic authentication to GnuPG.

Used software: OpenLDAP 2.2.27, run under SuSE 10.0 GnuPG 1.4.3rc1 (subversion revision 4020).

If you don’t want to wait until 1.4.3 is officially released, grab yourself a copy from svn:

> svn co svn://cvs.gnupg.org/gnupg/trunk


Attached is tarball with the files for OpenLDAP configuration, to which will be refered to below. I hope this doesn’t violate the rules of this list but the attachment is very small anyways.

You should have a basic understanding about LDAP first. If not, I’d recommend to read the OpenLDAP Admin Guide on http://www.openldap.org, which provides excellent documentation.

Also, as an LDAP client and excellent server management tool, I’d recommend phpLDAPadmin: http://phpldapadmin.sourceforge.net

The LDAP tree created in this example setup looks like:

  dc=EXAMPLE,dc=COM
  |
  +----cn=Manager
  +----cn=PGPServerInfo
  +----ou=PGP Keys
  |    +---pgpCertID=...
  |    +---pgpCertID=...
  +----ou=PGP Users
       +---uid=...
       +---uid=...

where dc=EXAMPLE,dc=COM is obviously the base DN.

First, install pgp-keyserver.schema from the tarball into to your schema directory. There are two more files which are not used here, but have been part of the schema I got from Peter, so I kept them for completeness.

Next, install slapd.conf and edit to suit your needs. That is, select either anonymous or user authentication.

In the provided file, anonymous writes are enabled. However, access is restricted to writes from localhost only. You may lift this restriction by modifying the peername.ip statement. See slapd.access(5) for details and examples.

Think twice before opening up anonymous writes, as _any_ user who can connect to your LDAP server can not only upload but also delete keys.

For user authentication, comment out update_anon and the access rule for anonymous writes. Users are stored as DN “uid=,ou=PGP Users,dc=EXAMPLE,dc=COM”.

You need to create users to bind to LDAP. One sample user is provided in ldif/pgpusers.ldif. Just copy the entry and modify it to create more and read the file to learn the used password.

Also, the password for the OpenLDAP manager is stored as a hash. It is ‘gpg’. Run slappasswd(8) to create a stronger password and replace the hash in slapd.conf.

Try to start your OpenLDAP server now. Under SuSE, I run “/etc/init.d/ldap start”.

Next, populate the directory with the basic layout by importing the example.ldif file (enter on a single line):

> cat example.ldif | ldapadd -x -W -h localhost
      -D "cn=Manager,dc=EXAMPLE,dc=COM"

When prompted for a password, enter the one you’ve created above or ‘gpg’ if you did not.

If you selected anonymous writes, you’re done configuring your OpenLDAP PGP keyserver.

If you selected user authentication, you need to add users now:

> cat pgpusers.ldif | ldapadd -x -W -h localhost
      -D "cn=Manager,dc=EXAMPLE,dc=COM"

Finally, you can use GnuPG to add keys (always on a single line):

For anonymous write:

> gpg --keyserver ldap://localhost --send-key 12345678


For user authentication (insecure on command-line, see below):

> gpg --keyserver ldap://localhost --keyserver-options
   "binddn=\"uid=user1,ou=PGPUsers,dc=EXAMPLE,dc=COM\""
   --keyserver-options bindpw=user1 --send-keys 12345678

To receive keys, simply do:

> gpg --keyserver ldap://localhost --recv-keys 12345678


Further notes:

* GnuPG looks for PGPServerInfo under the base DN.
  If you decide to put it somewhere else, use keyserver-option
  basedn to specify the new location, e.g.:
  keyserver-options "basedn=\"cn=PGPServerInfo,ou=PGP Info,dc=MYDOM\""

* Beware of shell quoting, like above which is the correct format
  if you  have spaces in your DN and specify the keyserver option
  on the command line.

* GnuPG can use TLS/SSL. For SSL, use ldaps:// and for tls the
  keyserver-options tls. It takes 'no','try','warn' or 'require'
  as an argument, e.g.:
  keyserver-options tls=require

* Put other keyserver options into ~/.gnupg/gpg.conf, e.g.:

  keyserver ldap://localhost
  keyserver-options binddn="uid=test1,ou=PGP Keys,dc=EXAMPLE,dc=COM"
  keyserver-options bindpw=verysecret
  keyserver-options tls=try
  keyserver-options verbose

  Then the following will just work:
  > gpg --send-keys 12345678
  or
  > gpg --recv-keys 12345678

* As it is INSECURE to specify your bind password on the command
  line, you should put it to your ~/.gnupg/gpg.conf and protect
  this file with 0600 permissions.

Well, that’s it for now. I hope this howto is helpful and somewhat complete! Good luck setting up your PGP keyserver with OpenLDAP.

I’d be glad if someone could verify the steps so that there are no glitches. Comments, notes, questions or else are appreciated.

Related files

List of related files and downloads

• /etc/debian_version
• /etc/lsb-release
• /etc/apt/sources.list Productive
• /etc/apt/sources.list Devel/Mixed
• /debian-cd/6.0.5/i386/iso-cd/debian-6.0.5-i386-netinst.iso

Debian Installation

Download the Debian Squeeze 6.x netinstall ISO.
If the version not exists, have a look at the netinstall website and choose the i386 from the “Small CDs” Menu.

Now boot with the ISO, click on the Install or Graphical Install menuoption and do the necesarry steps to prepare your partitions (recommended fs: ext3) and install the Debian Squeeze 6.x netinstall ISO base system. If you can choose from the list of available package compilations, just let the option base-system checked. After installing and rebooting to your new installation, you should have internet access and root access on the console, this is everything you need to perform the next steps.

Login as root (or “user”, but you will have sudo configured, which makes it more complicated).

1.0 Dist Upgrade

Now you’ll have to configure some basic conf files and to import the Ultimedia keyring which is needed for the signature checks that apt will perform.

  /etc/debian_version       (Update)
  /etc/lsb-release          (New file)
  /etc/apt/sources.list     (Move and recreate)

The following steps will upgrade the distribution and release on Debian systems:

1.1 /etc/debian_version

  $ echo "blade/unstable" > /etc/debian_version

1.2 /etc/lsb-release


$ echo "DISTRIB_ID=Ultimedia" > /etc/lsb-release
$ echo "DISTRIB_RELEASE=1.0.1" >> /etc/lsb-release
$ echo "DISTRIB_CODENAME=blade" >> /etc/lsb-release
$ echo "DISTRIB_DESCRIPTION=\"Ultimedia OS 1.0.1 (Blade)\"" >> /etc/lsb-release


1.3 /etc/apt/sources.list


$ mv /etc/apt/sources.list /etc/apt/~sources.list
$ echo "deb http://ftp.ultimediaos.com/ultimedia blade main contrib non-free" > /etc/apt/sources.list
$ echo "deb-src http://ftp.ultimediaos.com/ultimedia blade main contrib non-free" >> /etc/apt/sources.list


1.4 Keyring import and `apt dist-upgrade`

Download the keyring from our HTTP keyserver: server.

The simple way of doing this is reading the key into the STDIN stream and pipe it to apt-key that will read from STDIN or you could wget (download) it and add it manual.

  $ wget http://ks.ultimedia-linux.org/keyring/ultimedia-keyring.gpg \
    -O- | apt-key add -

Or in two steps:

  $ wget http://ks.ultimedia-linux.org/keyring/ultimedia-keyring.gpg

Import keyring to apt.

  $ apt-key add ultimedia-keyring.gpg

Update available packages, delete the old packages and perform the dist-upgrade.

  $ apt-get clean
  $ apt-get update
  $ apt-get dist-upgrade

During the upgrade it could be that the base-files package is replaced, this means that the before manual created files need to be upgraded. If debconf asks you how you’ll decide on the old configuration files, it is necesarry to press “Y”. It will upgrade the /etc/lsb-release and /etc/debian_version configuration again.

Install X11/Gnome2

The following steps will install X11 with gnome2 and GDM as display manager. You could also install Compiz Fusion but note that there is currently only the NVidia kernel driver builded, which will support only NVida graphic cards. The Xorg drivers mostly support all grafic cards but are not everytime able to use all features and modes (special with modern graphic cards), which makes it impossible to enable the compiz features. But feel free to experiment.

  $ apt-get install gnome
  $ reboot

Now your new bootloader will be grub2 and your new system is Ultimedia GNU/Linux OS.

Getting Started

First, install OpenSSH on two UNIX machines, hurly and burly. This works best using DSA keys and SSH2 by default as far as I can tell. All the other HOWTOs I’ve seen seem to deal with RSA keys and SSH1, and the instructions not surprisingly fail to work with SSH2.
On each machine type ssh somemachine.example.com and make a connection with your regular password. This will create a .ssh dir in your home directory with the proper perms.
On your primary machine where you want your secret keys to live (let’s say hurly), type


ssh-keygen -t dsa


This will prompt you for a secret passphrase. If this is your primary identity key, make sure to use a good passphrase. If this works right you will get two files called id_dsa and id_dsa.pub in your .ssh dir. Note: it is possible to just press the enter key when prompted for a passphrase, which will make a key with no passphrase. This is a Bad Idea ™ for an identity key, so don’t do it! See below for uses of keys without passphrases.

scp ~/.ssh/id_dsa.pub burly:.ssh/authorized_keys2

Copy the id_dsa.pub file to the other host’s .ssh dir with the name authorized_keys2.

Now burly is ready to accept your ssh key. How to tell it which keys to use? The ssh-add command will do it. For a test, type

ssh-agent sh -c 'ssh-add < /dev/null && bash'

This will start the ssh-agent, add your default identity(prompting you for your passphrase), and spawn a bash shell. From this new shell you should be able to:

ssh burly

This should let you in without typing a password or passphrase. Hooray! You can ssh and scp all you want from this bash shell and not have to type any password or passphrase.

Using X Windows

Now this is all well and good, but who wants to run their whole life from a single bash instance? If you use an X window system, you can type your passphrase once when you fire up X and all subprocesses will have your keys stored.

In the ~/.xinitrc file, modify your line which spawns windowmaker to read:

exec ssh-agent sh -c 'ssh-add < /dev/null && exec /usr/local/bin/wmaker

This will prompt you for your passphrase when you start up X, and then not again. All shells you spawn from X will have your keys stored.

This brings up a security issue- if someone comes upon your X session, they can spawn ssh sessions to burly and other hosts where you have put your id_dsa.pub information into the authorized_keys2 file. A locking screensaver like xlock is vital.

Different usernames

By default ssh assumes the same username on the remote machine. If you have a different username on the other machine, follow the normal ssh procedure:

[pkeck@hurly /]$ ssh -l paulkeck burly

More keys!

You are not limited to one public key in your authorized_keys2 file. Append as many as you like! If you, say, generated a unique private key on every machine you log into, and then appended the id_dsa.pub files to each of the other machines’ authorized_keys2 file, you’d have the equivalent of a .rhosts file with two added benefits:

Someone would need to know your passphrase to use it, so a cracker gaining access to an account on one machine will not jeopardize the other accounts. (If you foolishly use the same passphrase or, heaven forbid, id_dsa file on all the hosts, it would make it easier to exploit, so don’t do that.) Traffic is encrypted.

This command will do it without requiring an scp and vi session:

cat foo.pub |ssh burly 'sh -c "cat - >>~/.ssh/authorized_keys2"'

Single-purpose keys

So now you’re sshing and scping your brains out. Sooner or later you’ll come across one or both of these situations:

You want to automate some ssh/scp process to be done after hours, but can’t because no one will be around to type the passphrase. You want to allow an account to do some sort of ssh/scp operation on another machine, but are hesitant to append a key to your authorized_keys2 file because that essentially “opens the barn door” to anything that other account wants to do, not just the one operation you want to let it do. (This is the situation if you use a .shosts file.)

Single-purpose keys to the rescue!

Make yourself another key:

ssh-keygen -t dsa -f ~/.ssh/whoisit

ust press return when it asks you to assign it a passphrase- this will make a key with no passphrase required. If this works right you will get two files called whoisit and whoisit.pub in your .ssh dir.

cp ~/.ssh/whoisit.pub tempfile

We want to work on it a little. tempfile should consist of one really long line that looks kind of like this:

ssh-dss AAAAB3NzaC1k[...]9qE9BTfw== pkeck@hurly.example.com

Edit tempfile and prepend some things to that line so that it looks like this:

command="echo I\'m `/usr/ucb/whoami` on `/usr/bin/hostname`",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-dss AAAAB3NzaC1k[...]9qE9BTfw== whoisitnow

That will do what we want on Solaris; to try this example on Linux use this:

command="echo I\'m `/usr/bin/whoami` on `/bin/hostname`",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-dss AAAAB3NzaC1k[...]9qE9BTfw== whoisitnow

The stuff to prepend is your command that will be run when this key is activated, and some options to keep it from being abused (hopefully). The last thing on the line is just a comment, but you probably want to set it to something meaningful.

Also, most examples I see use no-pty as an additional option, but this messes up the carriage-return/linefeediness of the output of the above example. (Try it.) I haven’t looked into it enough to see why you would want it, but there you go.

cat tempfile |ssh burly 'sh -c "cat - >>~/.ssh/authorized_keys2"'

Append tempfile to your authorized_keys2 file on burly.

To “activate” (or perhaps “detonate”) the key from hurly (or anywhere that has the secret key), do this (maybe there is a better way?):

ssh -i ~/.ssh/whoisit burly

The following also works but is cumbersome:

ssh-agent sh -c 'ssh-add ~/.ssh/whoisit < /dev/null && ssh burly'

You can also append this “command key” to a different account’s authorized_keys2 file and trigger it from a different username. You just need the secret key. Like so:

ssh -i ~/.ssh/whoisit -l paulkeck burly'

The next leap in the pattern is something like this:

ssh -i /home/pkeck/.ssh/whoisit -l paulkeck burly'

This could be run by any user on the box if they could read your secret key, so always keep your .ssh dir and all your keys chmodded to 700 and 600 respectively.

You could make single-purpose keys with commands to (haven’t tested all these):

mt -f /dev/nst0 rewind

Rewind a tape on a remote machine

nice -n 19 dd of=/dev/nst0
Send STDIN to that tape drive. Maybe STDIN is a tar stream from tar -cf -.

nice -n 19 dd if=/dev/nst0
Read stuff from there to my STDIN

cat claxon.au > /dev/audio
Play an alarm noise on a remote machine

cat - > /dev/audio
Play any sound you send on STDIN

cat - > /etc/dhcpd.conf
Replace /etc/dhcpd.conf with some stuff from STDIN on the triggering machine (sounds like a temp file would be better)

You can send the stuff on STDIN with something like this on the triggering machine:

ssh-agent sh -c 'ssh-add ~/.ssh/whoisit < /dev/null && cat alarm.au | ssh burly'

or

ssh-agent sh -c 'ssh-add ~/.ssh/whoisit < /dev/null && tar cf - /home/pkeck | ssh burly'

Maybe for that one the corresponding command to “catch” that stream would be:

cat - > ~/backups/pkeck.tar.`date +%Y%m%d.%H-%M-%S`

You get the idea! Go crazy!

Tape examples from Ed Cashin’s Gettin’ Fancy with SSH Keys, my inspiration for getting into this whole thing!

Intro

Derivative Debian distributions vary in the domains of their specialization, user base, and the scale of modifications/extension they bring on top of vanilla Debian distribution. Therefore, there might not be a strict deterministic set of rules, and rather a set of guidelines could help to decide which actions should be taken by the developers of the derivative distribution to not inflict Debian, and, moreover, benefit from the Debian infrastructure/resources/frameworks where applicable.

Infrastructure

We encourage derivative distributions to mention and define their relationship with Debian on the web page that gives information about the derivative distro (usually the about page).
We encourage derivative distributions to use Debian infrastructure and the software that powers Debian infrastructure where possible.

Repositories

For those derivatives that re-use Debian binary packages, add some source packages and modify some source packages, where possible we encourage them to use standard Debian mirrors and add a second repository containing only the source and binary packages that have been added or modified.

For those derivatives that rebuild Debian source packages, add some source packages and modify some source packages, where possible we encourage them to use standard Debian mirrors for the source packages and add a second repository containing only the source and binary packages that have been added or modified. This recommendation may be hard to do and therefore regular source package syncing is an alternative recommendation.

Of course in both cases it is a good idea to run a Debian mirror to ensure source and binary availability. Any exact mirrors of Debian source and or binary packages should be registered with the Debian mirrors list.

If you are copying Debian source packages to your repositories without modifying them, please leave the signatures in place in the .dsc files, do not re-sign them with your own keys.

Keyrings

Please create your own keyring packages instead of patching the Debian keyring packages.

Releases

If your derivative is based on Debian stable releases, please start your release testing process at least when the Debian freeze happens or do regular release testing during the whole Debian release cycle.

Trademark

* Derivative distributions must not be named ”Debian”

* [[http://www.debian.org/trademark|Current Debian trademark policy]] states: To be fair to all businesses, we insist that no business use the name “Debian” in the name of the business, or a domain name of the business.
* ongoing work on the [[http://wiki.mako.cc/TrademarkFreedom|Draft of the new Debian trademark policy]] aims to clarify/relax above restriction. Consult the [[DebianProjectLeader]] meanwhile on a case-by-case basis

De-/Re-branding

Depending on the degree of divergence from the vanilla Debian, it might be necessary to introduce non-functional modifications in the deployed system to eliminate user confusion of the derivative distribution with vanilla ”Debian”

Entry points

Following packages along with corresponding files present users with ”Debian” name upon interaction with the system


* base-files
* /etc/issue
* /etc/issue.net
* /etc/dpkg/origins/default (symlink to distribution information file)
* /usr/share/base-files/motd
* grub-pc
* /etc/grub.d/05_debian_theme
* debian-installer
* package root/build/boot/x86/ (branding, references to 'Debian on all the fnumber screens')
* package root/build/config/local (override various strings)
* debian-cd
* will probably need customisation for the install images.
* synaptic
* uses the debian logo for indicating packages
* software-properties-gtk
* talks about dfsg and debian release (eg, squeeze). don't know if this counts


Artwork

* Any default background images carrying Debian official use logo?

* “Debian official use logo” with “Debian” word has usage restrictions: “This logo or a modified version may be used by anyone to refer to the Debian project, but does not indicate endorsement by the project. ”

Packages

Rebuilt Debian packages should carry distribution specific version suffix to avoid confusion with possibly API/ABI-incompatible original packages provided from Debian archives.

When modifying source packages, rename the Maintainer field to XSBC-Original-Maintainer and add a new Maintainer field.

Bug reports

Since Debian bug tracking system should not be used directly to report bugs in the derivative distributions, submitted bugreports should not be sent directly against Debian packages. “reportbug“ could either be switched (see /usr/share/doc/reportbug/README.developers.gz) or patched (please do not forget to make patch generic and submit it to Debian) to use some other bug tracking system/server; alternatively different address to submit reports could be specified per each source package in the Bugs field of source portion in “debian/control“ file.

Specific choice among above scenarios depends on the degree to which derivative distribution is changing/extending vanilla Debian system. For example, if the derivative does not introduce heavy reconfiguration of the stock Debian system, nor provides custom builds of non-leaf packages — it should be sufficient to provide custom Bugs: header fields only in rebuilt/new packages. If some base Debian libraries get customized/rebuilt and/or heavy re-configuration of the default Debian system in place, it is advised that all bugreports get submitted to the maintainers of the derivative distribution first for the analysis either the bug is pertinent to stock Debian, where it should be forwarded by the maintainers in such cases.

Popularity Contest

If you want to become the collector of popcon submissions, please do not simply divert popcon submissions from the default “popcon.debian.org“ to your server. Multiple target popcon servers could be listed in “SUBMITURLS“.

Benefits

* Debian would benefit from more adequate status on the usage of the work of Debian community
* If “apt“ package in the derivative carries custom suffix (since per se no other distribution-specific information is included in the popcon submissions) it could allow Debian to discover the most popular derivatives and provide some nice statistics of the usage beyond stock Debian
* Niche packages, which might not be very popular in stock Debian, could be more actively used in a specialized derivative distribution. Having adequate popcon statistic in Debian would guarantee that the package would not be removed from stock Debian, thus offloading maintenance burden on the interested derivative
* Debian’s popcon, unlike some other deployed popcon servers, might provide additional information (e.g. historical data) which might not be exposed on the derivative’s popcon website for some reason.

See also

* http://www.debian.org/logos
* http://wiki.mako.cc/TrademarkFreedom
* http://www.emdebian.org/emdebian/customisation.html

Language related it makes absolutly no sense to get listed on chinese searchspiders in my opinion, also it is another leaking point of security to get listed by unwanted searchengines. Crackers using they own searchalgorithms to scan for emails and services – and some providers makes it verry easy for them. Often those crackers occupy network ranges and bomb your servers, by sending emails over your smtp relay or they testing security breaches. Sometimes even the owners of the attacking machines did not know that they working on an occupied server and that they are member of a botnet.

This security issue could be solved by blocking the occupied ranges preemptively. To do so, you could configure the apache by checking the apache environment variables. Note that, if you have a vserver it would be recommended to block the ip ranges in routing levels. This could be done with ipables or route. See the linux manpages for more informations about the route command.

Apache 2 configuration (indirect blocking)

First of all, if you have full access to the webserver, build a global configuration file (i called it /etc/apache2/mod_security.conf) and put it to your conf.d path or load this file via the apache config Include directive, or you should build a .htaccess file if not (requires AllowOverride all).


# File: /etc/apache2/mod_security.conf

SetEnvIfNoCase User-Agent "^humanlinks" badbotlist
SetEnvIfNoCase User-Agent "^ia_archiver" badbotlist
SetEnvIfNoCase User-Agent "^IlseBot" badbotlist
SetEnvIfNoCase User-Agent "^Image\ Stripper" badbotlist
SetEnvIfNoCase User-Agent "^Image\ Sucker" badbotlist
SetEnvIfNoCase User-Agent "Indy\ Library" badbotlist
SetEnvIfNoCase User-Agent "^InfoNaviRobot" badbotlist
SetEnvIfNoCase User-Agent "^InfoTekies" badbotlist
SetEnvIfNoCase User-Agent "^Intelliseek" badbotlist
SetEnvIfNoCase User-Agent "^InterGET" badbotlist
SetEnvIfNoCase User-Agent "^Internet\ Ninja" badbotlist
SetEnvIfNoCase User-Agent "^Iria" badbotlist
SetEnvIfNoCase User-Agent "^Jakarta" badbotlist
SetEnvIfNoCase User-Agent "^JennyBot" badbotlist
SetEnvIfNoCase User-Agent "^JetCar" badbotlist
SetEnvIfNoCase User-Agent "^JOC" badbotlist
SetEnvIfNoCase User-Agent "^JustView" badbotlist
SetEnvIfNoCase User-Agent "^Jyxobot" badbotlist
SetEnvIfNoCase User-Agent "^Kenjin.Spider" badbotlist
SetEnvIfNoCase User-Agent "^Keyword.Density" badbotlist
SetEnvIfNoCase User-Agent "^larbin" badbotlist
SetEnvIfNoCase User-Agent "^LexiBot" badbotlist
SetEnvIfNoCase User-Agent "^lftp" badbotlist
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" badbotlist
SetEnvIfNoCase User-Agent "^likse" badbotlist
SetEnvIfNoCase User-Agent "^LinkextractorPro" badbotlist
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bo
SetEnvIfNoCase User-Agent "^LNSpiderguy" badbotlistt
SetEnvIfNoCase User-Agent "^LinkWalker" badbotlist
SetEnvIfNoCase User-Agent "^lwp-trivial" badbotlist
SetEnvIfNoCase User-Agent "^LWP::Simple" badbotlist
SetEnvIfNoCase User-Agent "^Magnet" badbotlist
SetEnvIfNoCase User-Agent "^Mag-Net" badbotlist
SetEnvIfNoCase User-Agent "^MarkWatch" badbotlist
SetEnvIfNoCase User-Agent "^Mass\ Downloader" badbotlist
SetEnvIfNoCase User-Agent "^Mata.Hari" badbotlist
SetEnvIfNoCase User-Agent "^Memo" badbotlist
SetEnvIfNoCase User-Agent "^Microsoft.URL" badbotlist
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" badbotlist
SetEnvIfNoCase User-Agent "^MIDown\ tool" badbotlist
SetEnvIfNoCase User-Agent "^MIIxpc" badbotlist
SetEnvIfNoCase User-Agent "^Mirror" badbotlist
SetEnvIfNoCase User-Agent "^Missigua\ Locator" badbotlist
SetEnvIfNoCase User-Agent "^Mister\ PiX" badbotlist
SetEnvIfNoCase User-Agent "^moget" badbotlist
SetEnvIfNoCase User-Agent "^NAMEPROTECT" badbotlist
SetEnvIfNoCase User-Agent "^Navroad" badbotlist
SetEnvIfNoCase User-Agent "^NearSite" badbotlist
SetEnvIfNoCase User-Agent "^NetAnts" badbotlist
SetEnvIfNoCase User-Agent "^NetMechanic" badbotlist
SetEnvIfNoCase User-Agent "^NetSpider" badbotlist
SetEnvIfNoCase User-Agent "^Net\ Vampire" badbotlist
SetEnvIfNoCase User-Agent "^NetZIP" badbotlist
SetEnvIfNoCase User-Agent "^NextGenSearchBot" badbotlist
SetEnvIfNoCase User-Agent "^NG" badbotlist
SetEnvIfNoCase User-Agent "^NICErsPRO" badbotlist
SetEnvIfNoCase User-Agent "^NimbleCrawler" badbotlist
SetEnvIfNoCase User-Agent "^Ninja" badbotlist
SetEnvIfNoCase User-Agent "^NPbot" badbotlist
SetEnvIfNoCase User-Agent "^Octopus" badbotlist
SetEnvIfNoCase User-Agent "^Offline\ Explorer" badbotlist
SetEnvIfNoCase User-Agent "^Offline\ Navigator" badbotlist
SetEnvIfNoCase User-Agent "^Openfind" badbotlist
SetEnvIfNoCase User-Agent "^OutfoxBot" badbotlist
SetEnvIfNoCase User-Agent "^PageGrabber" badbotlist
SetEnvIfNoCase User-Agent "^Papa\ Foto" badbotlist
SetEnvIfNoCase User-Agent "^pavuk" badbotlist
SetEnvIfNoCase User-Agent "^pcBrowser" badbotlist
SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" badbotlist
SetEnvIfNoCase User-Agent "^Pockey" badbotlist
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" badbotlist
SetEnvIfNoCase User-Agent "^ProWebWalker" badbotlist
SetEnvIfNoCase User-Agent "^psbot" badbotlist
SetEnvIfNoCase User-Agent "^Pump" badbotlist
SetEnvIfNoCase User-Agent "^QueryN.Metasearch" badbotlist
SetEnvIfNoCase User-Agent "^RealDownload" badbotlist
SetEnvIfNoCase User-Agent "Reaper" badbotlist
SetEnvIfNoCase User-Agent "Recorder" badbotlist
SetEnvIfNoCase User-Agent "^ReGet" badbotlist
SetEnvIfNoCase User-Agent "^RepoMonkey" badbotlist
SetEnvIfNoCase User-Agent "^RMA" badbotlist
SetEnvIfNoCase User-Agent "Siphon" badbotlist
SetEnvIfNoCase User-Agent "sitecheck.internetseer.com" badbotlist
SetEnvIfNoCase User-Agent "^SiteSnagger" badbotlist
SetEnvIfNoCase User-Agent "^SlySearch" badbotlist
SetEnvIfNoCase User-Agent "^SmartDownload" badbotlist
SetEnvIfNoCase User-Agent "^Snake" badbotlist
SetEnvIfNoCase User-Agent "^Snapbot" badbotlist
SetEnvIfNoCase User-Agent "^Snoopy" badbotlist
SetEnvIfNoCase User-Agent "^sogou" badbotlist
SetEnvIfNoCase User-Agent "^SpaceBison" badbotlist
SetEnvIfNoCase User-Agent "^SpankBot" badbotlist
SetEnvIfNoCase User-Agent "^spanner" badbotlist
SetEnvIfNoCase User-Agent "^Sqworm" badbotlist
SetEnvIfNoCase User-Agent "Stripper" badbotlist
SetEnvIfNoCase User-Agent "Sucker" badbotlist
SetEnvIfNoCase User-Agent "^SuperBot" badbotlist
SetEnvIfNoCase User-Agent "^SuperHTTP" badbotlist
SetEnvIfNoCase User-Agent "^Surfbot" badbotlist
SetEnvIfNoCase User-Agent "^suzuran" badbotlist
SetEnvIfNoCase User-Agent "^Szukacz/1.4" badbotlist
SetEnvIfNoCase User-Agent "^tAkeOut" badbotlist
SetEnvIfNoCase User-Agent "^Teleport" badbotlist
SetEnvIfNoCase User-Agent "^Telesoft" badbotlist
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" badbotlist
SetEnvIfNoCase User-Agent "^The.Intraformant" badbotlist
SetEnvIfNoCase User-Agent "^TheNomad" badbotlist
SetEnvIfNoCase User-Agent "^TightTwatBot" badbotlist
SetEnvIfNoCase User-Agent "^Titan" badbotlist
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" badbotlist
SetEnvIfNoCase User-Agent "^True_Robot" badbotlist
SetEnvIfNoCase User-Agent "^turingos" badbotlist
SetEnvIfNoCase User-Agent "^TurnitinBot" badbotlist
SetEnvIfNoCase User-Agent "^URLy.Warning" badbotlist
SetEnvIfNoCase User-Agent "^Vacuum" badbotlist
SetEnvIfNoCase User-Agent "^VCI" badbotlist
SetEnvIfNoCase User-Agent "^VoidEYE" badbotlist
SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" badbotlist
SetEnvIfNoCase User-Agent "^Web\ Sucker" badbotlist
SetEnvIfNoCase User-Agent "^WebAuto" badbotlist
SetEnvIfNoCase User-Agent "^WebBandit" badbotlist
SetEnvIfNoCase User-Agent "^Webclipping.com" badbotlist
SetEnvIfNoCase User-Agent "^WebCopier" badbotlist
SetEnvIfNoCase User-Agent "^webcollage" badbotlist
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" badbotlist
SetEnvIfNoCase User-Agent "^WebEnhancer" badbotlist
SetEnvIfNoCase User-Agent "^WebFetch" badbotlist
SetEnvIfNoCase User-Agent "^WebGo\ IS" badbotlist
SetEnvIfNoCase User-Agent "^Web.Image.Collector" badbotlist
SetEnvIfNoCase User-Agent "^WebLeacher" badbotlist
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" badbotlist
SetEnvIfNoCase User-Agent "^WebReaper" badbotlist
SetEnvIfNoCase User-Agent "^WebSauger" badbotlist
SetEnvIfNoCase User-Agent "^WebSite" badbotlist
SetEnvIfNoCase User-Agent "^Website\ eXtractor" badbotlist
SetEnvIfNoCase User-Agent "^Website\ Quester" badbotlist
SetEnvIfNoCase User-Agent "^Webster" badbotlist
SetEnvIfNoCase User-Agent "^WebStripper" badbotlist
SetEnvIfNoCase User-Agent "^WebWhacker" badbotlist
SetEnvIfNoCase User-Agent "^WebZIP" badbotlist
SetEnvIfNoCase User-Agent "^Wget" badbotlist
SetEnvIfNoCase User-Agent "Whacker" badbotlist
SetEnvIfNoCase User-Agent "^Widow" badbotlist
SetEnvIfNoCase User-Agent "^WISENutbot" badbotlist
SetEnvIfNoCase User-Agent "^WWWOFFLE" badbotlist
SetEnvIfNoCase User-Agent "^WWW-Collector-E" badbotlist
SetEnvIfNoCase User-Agent "^Xaldon" badbotlist
SetEnvIfNoCase User-Agent "^Xenu" badbotlist
SetEnvIfNoCase user-Agent "YandexBot" badbotlist
SetEnvIfNoCase User-Agent "^Zeus" badbotlist
SetEnvIfNoCase User-Agent "^Zyborg" badbotlist
SetEnvIfNoCase User-Agent "ZmEu" badbotlist
<IfModule mod_setenvif.c>


To load this file from the main httpd.conf, open it and put this line into the configuration file. Note that the file is autoloaded if apache was configured to use a conf.d path.


# XXX: Implemented mod_security
Include /etc/apache2/mod_security.conf


Now open your vhost configuration of the website you want to protect against the botlist. The lines look like this:


<Directory /www/vhosts/example.com/httpdocs>
<Files *>
Order allow,deny
Allow from all
Deny from env=badbotlist
</Files>
</Directory>


and restart the webserver: apache2ctl graceful (on Debian based systems). On openSuSE you could use: rcapache2 graceful. Or simply use /etc/init.d/apache2 restart|reload.

Explicit blocking in the Kernel routing tables

During the fight against the bots, i was so frustrated that i blocked the ip ranges for a short time. The list below shows verry disturbing networks. Decide by yourself which of them to block. Note that this could be only the last solution if really nothing of the bad bot blocks will work. But after some weeks, the bots are gone.


61.51.16.0 - 61.51.31.255 61.51.16.0/20 # (Baidu China - Beijing)
14.136.0.0 - 14.136.255.255 14.136.0.0/16 # (Baidu China - H.K.)
123.125.71.0 - 123.125.71.255 123.125.71.0/24 # (Baidu China)
14.208.0.0 - 14.223.255.255 14.208.0.0/12 # (Baidu China)
95.108.241.0 - 95.108.241.255 95.108.241.0 # (YandexBot Russian Federation)
95.108.151.0 - 95.108.151.255 95.108.151.0 # (YandexBot Russian Federation)
119.63.192.0 - 119.63.199.255 119.63.192.0/21 # (Baidu Japan Inc.)
119.63.192.0 - 119.63.199.255 119.63.196.0/24 # (Baidu Japan Inc.)
180.76.0.0 - 180.76.255.255 180.76.0.0/16 # (Baidu China, Baidu Plaza, Beijing)
220.181.0.0 - 220.181.255.255 220.181.108.0/24 # (CHINANET Beijing Province Network)
123.125.71.0 - 123.125.71.255 123.125.71.0/24 # (Baidu China)
202.46.32.0 - 202.46.63.255 202.46.32.0/19 # (Baidu China)
39.112.0.0 - 39.127.255.255 39.112.0.0/12 # KOREAN
211.148.192.0 - 211.148.223.255 211.148.192.0/19 # China ShenZhen Topway Video Communication Co. Ltd.
58.208.0.0 - 58.223.255.255 58.208.0.0/12 # CHINANET jiangsu province network
117.79.128.0 - 117.79.191.255 117.79.128.0/18 # Beijing Sanxin Shidai Co.Ltd (denial of time protocol)
122.225.11.0 - 122.225.11.255 122.225.11.0 # CHINANET


Lets say you want to block an address from above. Issue the following command:


/usr/sbin/iptables -A INPUT -p udp -s 61.51.16.0/20 -j DROP
/usr/sbin/iptables -A INPUT -p tcp -s 61.51.16.0/20 -j DROP


That’s it for the first time. Maybe if i have the time, i will extend this section with a tutorial about ip6tables. Also a verry good solution im using is fail2ban which i’ll introduce during my next posts.

– Fight the freedom but keep the peace.

If you are reading this article keep in mind that content marked by * was written two years ago and pasted into this blog. This means that some applications may have changed in its usage or the way of doing is not always common.

Shell magics and internals

A reverse search to often used commands could be verry useful. Bash provides it and it is called reverse-i-search. You could initialize that mode by pressing the following combination:

ctrl + R

This will change into the reverse search mode and will pop up:

   (reverse-i-search)`': 

Now if you type the string: apt-g, you’ll see the last typed command that is matching. Note that everything depends on the commands you have used before:

   (reverse-i-search)`apt-g': apt-get remove gdm

If you press the ctrl – R again you’ll be able to walk thru all results the bash will find for that string.

The argument trick

Maybe you have pressed it allready, but you did not know what it is good for, it is the arg editing mode. I call it this way, please correct me if this is not the right name.

On the console type:
$ echo he

Now you type the double “ll” and “o”, OR you could perform the following trick:

Press:
alt + 2 + l

  (arg: 2) l

this will give back a double “ll” to the prompt. In this case it looks like this:

$ echo hell

But, this is not useful if you have short strings like that and maybe only useful if you want to type a high ammount of arguments (test data) in a short time.

Process handling

Sometimes it is useful to send a process in the background to start another or because it is unnecesarry to se the whole output. To actually send a process to the background press:

ctrl + Z
[1]+ Stopped top

Now the process is running in the background and you could type:

fg

to bring the process back to foreground. Now lets assume you try to start a second process and you’ve got to select between them. To do so type:

jobs
[1]- Stopped top
[2]+ Stopped watch -n2 ps auxf

Now bring the process 1 which is the top command to the forgeround.
fg 1

This will bring back the top process to the forground and you will see the output is going back to stdout.

Buffer stdout stream

If you want to buffer the whole stdout stream, this is also possible. Often this is pressed by beginners and will end in frustrating ending of the whole terminal session.

Start an application which actually sends content into the background, for instance:

$ sleep 10; df;

After executing this command interruppt the stdout stream by pressing the combination alt +s. Ok, now you have interupted the stdout stream and after drinking a cup of coffe and maybe going for a walk you could press alt +q which will release the whole buffered stdout stream.

Find Files by Extensions and do something with it *

In some cases you’ll want to search the harddrives for files by extension and do some actions with them. In this example i’ll show the usage of the find command. Lets assume i want to md5sum some files by extension but only img iso files and zip archives.

   for item in `find . -regex ".*\.\(img\|zip\)"`; do 
      md5sum $item; 
   done;

Convert PDF to image JPEG *

To convert from PDF format to JPEG format you can use the convert command [CONVERT(1)]. The convert command is a powerfull ImageMagick command. Better have a look into the manual man convert[CONVERT(1)] before you go ahead.

This command will convert an entire PDF document.

 /usr/bin/convert file.pdf -geometry 120 -quality 70 jpg:file.jpg

If you’ll need a small preview image from an entire PDF document, you should consider using another approach to get a faster and better result. In this case the approach ist GhostScript. For the optimizing features you’ll have to look inside the manual for the GhostScript (gs) command. man gs [GS(1)]

  /usr/bin/gs -q 
  -dBATCH                  \ 
  -dMaxBitmap=300000000    \ 
  -dNOPAUSE                \ 
  -dSAFER                  \ 
  -sDEVICE=jpeg            \ 
  -dTextAlphaBits=4        \  
  -dGraphicsAlphaBits=4    \ 
  -dFirstPage=1            \
  -dLastPage=1             \ 
  -sOutputFile=./test.jpg  \
  ./test.pdf -c quit

Find any text phrase in Files by Directory *

Sometimes you just want to search in directories for a sepcific text phrase. In this case there are some usefull console commands. The first approach is very simple and gives back only the found text phrase.

 
  cat * | grep -H -n -i searchphrase

This command is suboptimal in case that cat * will give back all files as a string, so grep could not return the right line and filename. In case about that it is better to call only the grep command like this:

 
  grep -H -n -i searchphrase ./*

For recursive search requests add the -R option.

  grep -R -H -n -i searchphrase ./*

The third and much better approach for advanced scripting purposes is a for loop. This gives you more possibilties for defining own actions that you want to execute in case you found a text phrase. In my sample the script returns back the filename where i found the text phrase. This action could been extended and combined with other scripts.

 for FILE in *; do
    if [ "$(cat ${FILE} | grep -i searchphrase)" != "" ]; then
      echo ${FILE}
    fi
 done

Search for files by extension and create a filelist *

Following command will create a filelist by using a for loop and the find command.

 for filename in `find your/path/  -name '*.js' `; \ 
     do echo -e "file:${filename}" >> file.lst; done; \
 for filename in `find my/path/  -name '*.txt' `; \ 
     do echo -e "file:${filename}" >> file.lst; done;

Using AWK to replace a Unix Timestamp in multiline messages

If an application uses the unix-timestamp prefixed in logfiles, it is verry hard to read at which time the event was loged. So for this example im using the awk functions to replace the timestamp and calculate it to a readable Date/Time format. Lets see the script:

If $msg holds the complete logfile content, i assume this solution. Have a look at the comments, they describe what we are able to use for replacements.

    msg="`echo -e -n "${msg}" | awk NF`" # strip out empty lines
    msg="`echo -e -n "${msg}"| awk -F"\n" '
        # Returns a string in the format of output of date(1)
        # Populates the array argument time with individual values:
        #    time["second"]       -- seconds (0 - 59)
        #    time["minute"]       -- minutes (0 - 59)
        #    time["hour"]         -- hours (0 - 23)
        #    time["althour"]      -- hours (0 - 12)
        #    time["monthday"]     -- day of month (1 - 31)
        #    time["month"]        -- month of year (1 - 12)
        #    time["monthname"]    -- name of the month
        #    time["shortmonth"]   -- short name of the month
        #    time["year"]         -- year modulo 100 (0 - 99)
        #    time["fullyear"]     -- full year
        #    time["weekday"]      -- day of week (Sunday = 0)
        #    time["altweekday"]   -- day of week (Monday = 0)
        #    time["dayname"]      -- name of weekday
        #    time["shortdayname"] -- short name of weekday
        #    time["yearday"]      -- day of year (0 - 365)
        #    time["timezone"]     -- abbreviation of timezone name
        #    time["ampm"]         -- AM or PM designation
        #    time["weeknum"]      -- week number, Sunday first day
        #    time["altweeknum"]   -- week number, Monday first day

        function gettimeofday(time, ret, now, i) {
         # get time once, avoids unnecessary system calls
         now = systime()

         # return date(1)-style output
         ret = strftime("%a %b %d %H:%M:%S %Z %Y", now)

         # clear out target array
         delete time

         # fill in values, force numeric values to be
         # numeric by adding 0
         time["second"]       = strftime("%S", now) + 0
         time["minute"]       = strftime("%M", now) + 0
         time["hour"]         = strftime("%H", now) + 0
         time["althour"]      = strftime("%I", now) + 0
         time["monthday"]     = strftime("%d", now) + 0
         time["month"]        = strftime("%m", now) + 0
         time["monthname"]    = strftime("%B", now)
         time["shortmonth"]   = strftime("%b", now)
         time["year"]         = strftime("%y", now) + 0
         time["fullyear"]     = strftime("%Y", now) + 0
         time["weekday"]      = strftime("%w", now) + 0
         time["altweekday"]   = strftime("%u", now) + 0
         time["dayname"]      = strftime("%A", now)
         time["shortdayname"] = strftime("%a", now)
         time["yearday"]      = strftime("%j", now) + 0
         time["timezone"]     = strftime("%Z", now)
         time["ampm"]         = strftime("%p", now)
         time["weeknum"]      = strftime("%U", now) + 0
         time["altweeknum"]   = strftime("%W", now) + 0
         time["isodate"]      = strftime("%F", now)
         time["isotime"]      = strftime("%T", now)

         return ret
        }
        BEGIN { 
            FS="[\n]" 
            gettimeofday(now)
            datestring=now["isodate"]"-"now["isotime"]
        }
        { 
            for (i=1; i<=NF; i++) {
                printf "%s: %s\n", datestring,tolower($i)
            } 
        }
    '`"

Useful alias definitions

For network admins this alias definitions may be useful? Try them.

alias ns="netstat -oNplWna"  # netstat
alias nse="netstat -a | grep ESTABLISHED" #netstat established
alias nsw="watch -n2 'netstat -a | grep ESTABLISHED'" # netstat watch
alias nsi="netstat -iW" # netstat interfaces
alias nsr="netstat -rW" # netstat routing
alias nsg="netstat -gW" # netstat ipv4/ipv6 group membership

# watch logs
alias wl="watch -n 2 '/usr/bin/tail /var/log/apache2/access_log && /usr/bin/tail \
          /var/log/apache2/error.log && /usr/bin/tail /var/log/messages && /usr/bin/tail \
          /var/log/syslog && /usr/bin/tail /var/log/auth.log'"

# or as shorter example + sudo  (keep an eye what logfiles are present!)
alias wl='sudo /usr/bin/tail -f -n 4 /var/log/{auth.log,daemon.log,debug,messages,mysql,syslog,user.log}'

Listen on serial lines via uucp

A common gotcha is the following scenario. You try to open the serial line and you can't remember the line settings and/or application usage. Sometimes it's possible that the device is not readable by root and only by tty (add yourself to this group or change the mode). This is the ideal time to create some alias definitions for us. Requirements to communicate with the tty is cu (uucp).

Alias  Description
---------------------------------
s0     Listen to /dev/ttyS0
s1     Listen to /dev/ttyS1
u0     Listen to /dev/ttyUSB0

In the example source im using sudo to obtain a shell with root permissions. In this root shell the chown command will apply the correct user and group for the current serial tty. If this succeed the command cu will be executed with the default settings. Note that you could use the same code also as root.

# The values could also be defined by hand in the alias definitions below, 
#   just replace the $TTYLS variable with the required line speed.
export TTYLS=115200 # tty line speed

# Alias s0 for serial port 1
if [ -c /dev/ttyS0 ]; then
    alias s0="/usr/bin/sudo /bin/sh -c '/bin/chown root:root /dev/ttyS1 && /usr/bin/cu -s $TTYLS -l ttyS1'"
fi;

# Alias s1 for serial port 1
if [ -c /dev/ttyS1 ]; then
    alias s1="/usr/bin/sudo /bin/sh -c '/bin/chown root:root /dev/ttyS1 && /usr/bin/cu -s $TTYLS -l ttyS1'"
fi;

# Alias u0 for USB/Serial port 1
if [ -c /dev/ttyUSB0 ]; then
    alias u0="/usr/bin/sudo /bin/sh -c '/bin/chown root:root /dev/ttyUSB0 && /usr/bin/cu -s $TTYLS -l ttyUSB0'"
fi;

Random Examples

1. Obtain a root shell, switch back to your current $USER and perform the make - and as superuser shutdown machine after all.

sudo sh -c "su -c 'make' $USER && init 0"

2. Show a detailed netstat with Proto, Local Address, Foreign Address and PID/Program name.

netstat -tanp

3. Send a directory via scp protocol to a remote/local location.

scp -r source@local:/directory_to_copy dest@remote.or.local:/pathwhereto/

Note: Ultimedia dpkg-extension was not intended to use in the community, it is a verry speacial script working on ultimedia/blade build systems.

• Defines a maintainer profile and includes verry related developer settings
• Defines a default dist profile and archive related settings to sync and upload and re/backup
• Search thru the Ultimedia archive and package apt-cache
• Parse debian/watch files (udeh)
• Close bugs with dpkg_close
• Sign *.changes files
• Cleanup buildsources
• Obtain build-deps inside the build folder with dpkg_deb
• Execute debchange with whiptail powered gui
• Download sources, resolve dependencies, extract source, build and sign with dpkg_apt_source
• Maintain the Ultimedia archive

1. Login and Build Process
2. Little Helper commands

• Package Development Server: 10.1.2.254
• Package Development Path: /debian/source/[username]
• Scripts: /etc/profile.d/

1. Note: Since using sudo it is no more needed to obtain root rights by su, just login as user.
2. Note: As user it is possible you are not able to access the lockfile database and you will get this message:

Error 13 creating lock file '/srv/www/apt/repodata/ultimedia/db/lockfile': Permission denied!
There have been errors!
If performing "dpkg_search" or "dpkg_apt_source" you can save ignore this message!


This Example converts/includes autogen-5.10 into the Ultimedia
GNU/Linux package repository and shows the usage of
/etc/profile.d/dpkg-extension.sh. For further informations about the
commands you can type “man dpkg-extension.sh”.

Login Process

• Login as root on “Console A” and login as [user]
  on “Console B”.
• Update APT Package Database (if updates required in the core
  packages, perform them first).
• Core Packages always need to be builded before you
  build/synchronize a Debian Squeeze Package.

On Console A:

$ apt-get update


• Search for the Source Package name (inside the Ultimedia and
  Debian Package Database)

On Console A:

$ dpkg_search libopts25

I: no packages found
I: source package is autogen


The output message means, that the Package is not available inside the Ultimedia Package Database and the Source Package name is autogen.

• Download Source Package from Debian Package Server

On Console B:

$ dpkg_apt_source autogen no

Error 13 creating lock file '/srv/www/apt/repodata/ultimedia/db/lockfile': Permission denied!
There have been errors!
mkdir: created directory `/debian/source/markus/autogen'
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'autogen' packaging is maintained in the 'Git' version control system at:

Need to get 1400 kB of source archives.
Get:1 http://ftp.de.debian.org/debian/ squeeze/main autogen 1:5.10-1.1 (dsc) [1884 B]
Get:2 http://ftp.de.debian.org/debian/ squeeze/main autogen 1:5.10-1.1 (tar) [1385 kB]
Get:3 http://ftp.de.debian.org/debian/ squeeze/main autogen 1:5.10-1.1 (diff) [12.9 kB]
Fetched 1400 kB in 1s (1115 kB/s)
Download complete and in download only mode
gpgv: keyblock resource `/home/markus/.gnupg/trustedkeys.gpg': file open error
gpgv: Signature made Sun Jan 31 00:26:53 2010 CET using RSA key ID 8313B5F0
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./autogen_5.10-1.1.dsc
dpkg-source: info: extracting autogen in autogen-5.10
dpkg-source: info: unpacking autogen_5.10.orig.tar.gz
dpkg-source: info: applying autogen_5.10-1.1.diff.gz
dpkg-source: info: upstream files that have been modified:
autogen-5.10/autoopts/autoopts-config.in

Source: autogen
Version: 1:5.10-1.1
Distribution: unstable
Urgency: low
Maintainer: Kurt Roeckx <kurt@roeckx.be>
Date: Sat, 30 Jan 2010 20:13:56 +0100
Closes: 562607 562791
Changes:
autogen (1:5.10-1.1) unstable; urgency=low
.
* Non-maintainer upload.
* Set shlibs so that other packages get proper Depends (Closes: #562791)
* Don't let autoopts-config return an rpath. This is not useful on any
Debian system. (Closes: #562607)


The above command will create the directory “autogen” (if not exists) chdir inside autogen, downloads the Source Package from the Debian Servers, extracts it and chdir inside the extracted folder. The no switch at the end will disable the direct start of the build process as you not have incremented the Version number. The gpg error (gpgv: keyblock resource `/home/markus/.gnupg/trustedkeys.gpg’: file open error) indicates that there are public keys missing on the build system, note to install the debian-maintainers keyring or you’re unable to verrify the source package, which is not allowed in the Debian Policy.

Note: There exists an alias for dpkg-parsechangelog: cl.

• Update, increment or add Ultimedia/blade version suffix to
  current version.

On Console B:

$ dch --newversion=1:5.10-1.1ultimedia1 Synchronized with Debian Source


Check the new version with:

$ cl


or

$ dpkg-parsechangelog

Source: autogen
Version: 1:5.10-1.1ultimedia1
Distribution: unstable
Urgency: low
Maintainer: Markus Niewerth <mniewerth@ultimediaos.com>
Date: Fri, 30 Mar 2012 01:18:31 +0200
Changes:
autogen (1:5.10-1.1ultimedia1) unstable; urgency=low
.
* Synchronized with Debian Source


Notes to dfsg versions:
(dch warning: no orig tarball found for the new version.)

If we have a version like this: ntp_4.2.6.p2+dfsg-1, we won’t increment the Debian Free Software Guidelines TAG, so we remove this TAG from the Ultimedia release.

On Console B:

$ dch --newversion=1:4.2.6.p2+ultimedia-1 Synchronized with Debian source
dch warning: your current directory has been renamed to:
../ntp-4.2.6.p2+ultimedia
dch warning: no orig tarball found for the new version.


This warning appears because the original Tarball matches not our current version. A directory listing shows the issue we currently dealing with.

$ l
total 4125
drwxr-xr-x 24 markus markus 1840 Mar 29 16:17 ntp-4.2.6.p2+ultimedia
-rw-r--r-- 1 markus markus 3793444 Jul 13 2010 ntp_4.2.6.p2+dfsg.orig.tar.gz
-rw-r--r-- 1 markus markus 415786 Jul 13 2010 ntp_4.2.6.p2+dfsg-1.debian.tar.gz


The easy way to create the original Tarball with the Ultimedia version TAG, is to rename the current
ntp_4.2.6.p2+dfsg.orig.tar.gz
to
ntp_4.2.6.p2+ultimedia.orig.tar.gz
.

$ mv ntp_4.2.6.p2+dfsg.orig.tar.gz ntp_4.2.6.p2+ultimedia.orig.tar.gz
$ cd ntp-4.2.6.p2+ultimedia/
...
($ dpkg_build)


• The Control Files in folder: debian/.
  • Open the file debian/control with gedit as a user not as
    root and edit the following lines.

Source: autogen
Section: devel
Priority: optional
Maintainer: Bradley Smith <bradsmith@debian.org>
Build-Depends: debhelper (>= 7), autotools-dev, gperf, guile-1.8-dev, libxml2-dev, texinfo, texlive, texi2html, quilt
Standards-Version: 3.8.3
Homepage: http://www.gnu.org/software/autogen/
Vcs-Git: http://git.brad-smith.co.uk/git/debian/pkg-autogen.git
Vcs-Browser: http://git.brad-smith.co.uk/?p=debian/pkg-autogen.git


In this control file the developers used a Git repository. Remove the field Vcs-Git and replace with Vcs-Svn as we now only using SubVersion as VCS system. Change the field Maintainer and if used the Uploaders field. Move on with the next step: Create a subversion repository.

This fields should be updated:

1. Maintainer
2. Uploaders
3. Vcs-Browser
4. Vcs-Svn

Don’t forget to rename the Maintainer field respectivly into XSBC-Original-Maintainer according the Debian Derivate Guidlines (Census).

In the autogen example i’ve changed the following fields:

1. Maintainer: Markus Niewerth <mniewerth@ultimediaos.com>
2. XSBC-Original-Maintainer: Bradley Smith <bradsmith@debian.org>
3. Vcs-Svn: http://svn.ultimediaos.com/svn/pkg-autogen/trunk/
4. Vcs-Browser:

   http://svn.ultimediaos.com/wsvn/pkg-autogen/trunk/

See the Debian Developer Documentation for more infos about the
control files.

• Create a logentry inside the debian/changelog file with:

$ dpkg_edit_cl


• Create a close logentry inside debian/changelog file with:

$ dpkg_close [BUGID]


• Create a Subversion Repository.

This step requires root access on the production server, if you are not an Admin of the Ultimedia Version Control Systems, you might request the creation of a repository and move on with: Create a Subversion Project.

1. Login as root and do the following steps to create the source
   storage.

$ cd /srv/vcs/source/
$ mkdir pkg-autogen
$ cd pkg-autogen


1. Create the SubVersion Repository with UltimediaSVN usvn.

$ usvn -c pkg-autogen

DBG: #981 svnadmin returns: 1
DBG: #867 ShellCMD: /usr/bin/svn import /tmp/vcs-tmp file:///srv/vcs/svn/pkg-autogen --message "Initial structure import."
DBG: #871 Current Working Directory: /srv/vcs/source/pkg-autogen
Adding /tmp/vcs-tmp/trunk
Adding /tmp/vcs-tmp/branches
Adding /tmp/vcs-tmp/tags

Committed revision 1.

$ usvn -ac pkg-autogen

I: Restarting webserver...
Syntax OK


Now you created the SubVersion repository and you are able to administer the repository structure. Im personally using Eclipse with Subversive SVN connectors, which makes an administration and working verry easy.

The URL to the new repository should be:
http://svn.ultimediaos.com/svn/pkg-autogen

You will need a Developer account to have access to the subversion project. Remove the old TTB structure inside the repository, (only if you have a MULTI PROJECT) with the following log message:

Deleting old obsolete project layout.


Create a new Project Structure inside the repository, for
instance autogen. (In our example it is not needed!) Use this log
message if you create a new project inside an existing subversion
repository.

Creating new project layout: pkg-autogen/my-project


Go back to edit the control files in Step: The Control Files in folder: debian/, and edit the following fields. This URLS may be used for Vcs-Browser and Vcs-Svn control fields:

Vcs-Svn: http://svn.ultimediaos.com/svn/pkg-autogen/trunk/
Vcs-Browser: http://svn.ultimediaos.com/wsvn/pkg-autogen/trunk/


• Start the build process.

Now you could try a first build as user by using the following
command:

On Console A:

$ apt-get build-dep autogen


On Console B:

$ dpkg_build


Sign the package with your username/password and your certificate which was obtained from the Ultimedia Key Server. See the GPG2 section for more informations about the keyservers.

signfile autogen_5.10-1.1ultimedia1.dsc

You need a passphrase to unlock the secret key for
user: "Markus Niewerth <mniewerth@ultimediaos.com>"
1024-bit DSA key, ID C27C484F, created 2011-07-22

dpkg-genchanges >../autogen_5.10-1.1ultimedia1_i386.changes
dpkg-genchanges: not including original source code in upload
signfile autogen_5.10-1.1ultimedia1_i386.changes

You need a passphrase to unlock the secret key for
user: "Markus Niewerth <mniewerth@ultimediaos.com>"
1024-bit DSA key, ID C27C484F, created 2011-07-22

dpkg-source --after-build autogen-5.10
dpkg-buildpackage: binary and diff upload (original source NOT included)


Now the package was builded and you can cleanup the package and
build a TARBALL for the Subversion Repository.

On Console A:

$ dpkg_clean_source


$ cd .. $ tar cjf autogen-5.10.tar.bz2 autogen-5.10 $ scp
autogen-5.10.tar.bz2 root@ultimediaos.com:/srv/vcs/source/pkg-autogen/

Now extract the source on the package server and import the
source into the subversion repository.

$ cd /srv/vcs/source/pkg-autogen
$ tar xf autogen-5.10.tar.bz2
$ cd


On Console B:

$ svn import . http://svn.example.com/svn/pkg-autogen/trunk --message "Initial source import"

Adding ….

• Maintainer Upload.

Now it is needed to publish the source and debian packages to
the Ultimedia package server.

On Console B:

$ mkdir build
$ mv autogen_*.{gz,dsc,changes} build
$ mv *.deb build
$ cd build
$ l

total 2542
-rw-r--r-- 1 markus markus 1385329 Nov 17 2009 autogen_5.10.orig.tar.gz
-rw-r--r-- 1 markus markus 13227 Mar 30 23:07 autogen_5.10-1.1ultimedia1.diff.gz
-rw-r--r-- 1 markus markus 1283 Mar 30 23:13 autogen_5.10-1.1ultimedia1.dsc
-rw-r--r-- 1 markus markus 2328 Mar 30 23:13 autogen_5.10-1.1ultimedia1_i386.changes
-rw-r--r-- 1 markus markus 1018232 Mar 30 23:11 autogen_5.10-1.1ultimedia1_i386.deb
-rw-r--r-- 1 markus markus 101348 Mar 30 23:11 libopts25-dev_5.10-1.1ultimedia1_i386.deb
-rw-r--r-- 1 markus markus 62718 Mar 30 23:11 libopts25_5.10-1.1ultimedia1_i386.deb

$ dpkg_include

autogen_5.10-1.1ultimedia1.dsc: component guessed as 'main'
I: changelog upload...
I: changelog uploaded to: /srv/www/apt/repodata/changelogs/pool/main/a/autogen/autogen_1:5.10-1.1ultimedia1/
Created directory "/srv/www/apt/repodata/ultimedia/pool/main/a/autogen"
Exporting indices...
Successfully created '/srv/www/apt/repodata/ultimedia/dists/blade/Release.gpg.new'
Successfully created '/srv/www/apt/repodata/ultimedia/dists/blade/InRelease.new'
Skipping inclusion of 'autogen' '1:5.10-1.1ultimedia1' in 'blade|main|source', as it has already '1:5.10-1.1ultimedia1'.
Exporting indices...
Successfully created '/srv/www/apt/repodata/ultimedia/dists/blade/Release.gpg.new'
Successfully created '/srv/www/apt/repodata/ultimedia/dists/blade/InRelease.new'


Now sync the local package and changelog repository with the
online repository. (root password required)

$ dpkg_sync

dpkg_sync: Variable setup ...
dpkg_sync: source_dir=/srv/www/apt/repodata/ultimedia/
dpkg_sync: destin_dir=upackage@10.1.2.254:/srv/www/vhosts/packages.ultimedia-box.de/httpdocs/ultimedia/
dpkg_sync: excludes=
dpkg_sync: rsync_params=-azv --progress --delete
dpkg_sync: Full command-line ...
/usr/bin/rsync -azv --progress --delete \
/srv/www/apt/repodata/ultimedia/ \
upackage@10.1.2.254:/srv/www/vhosts/packages.ultimedia-box.de/httpdocs/ultimedia/ \

Password:


Now the package is uploaded and the servers are in sync. You should now do a apt-get update on the Ultimedia system.

Little Helper commands:

• SVN import (inconsistent newlines)

If you are troubled with a message on “svn import” like this:

svn: File 'html/drivers/driver44.html' has inconsistent newlines
svn: Inconsistent line ending style


The solution to this issue is dos2unix and the conversion of inconsitent files.

$ ultimedia # find my-project-folder -name "*" -type f | xargs dos2unix


• Create a new debian/changelog file

The following command should be used to create a new debian/changelog file for an initial release. Note: the package example is uucp-1.07-20blade1

$ ultimedia:/src/uucp/uucp-1.07 # mv debian/changelog ../../~debian.changelog
$ ultimedia:/src/uucp/uucp-1.07 # dch --create --package=uucp --newversion=1.07-20blade1 --distribution=unstable Initial release.


Update a Non-Maintainer-Upload package

A Non-Maintainer-Upload version should be raised/updated this way:

Note: the package example is po-debconf (1.0.16+nmu1)

$ ultimedia:/src/po-debconf/po-debconf-1.0.16+nmu1] # mv debian/changelog ../../~debian.changelog
$ ultimedia:/src/po-debconf/po-debconf-1.0.16+nmu1] # dch --newversion=1.0.16+nmu2ultimedia1 \
                                                      --distribution=unstable Merged with Debian squeeze.

dch warning: your current directory has been renamed to:
../po-debconf-1.0.16+nmu2ultimedia1
dch warning: no orig tarball found for the new version.

$ ultimedia:/src/po-debconf/po-debconf-1.0.16+nmu2ultimedia1] # mv ../po-debconf_1.0.16+nmu1.tar.gz \
                                                                ../po-debconf_1.0.16+nmu2ultimedia1.tar.gz