A server having 2 disks, but running on single disk (history..) which has alsready some bad SMART values...
So, I've changed the server "on the fly" to a raid-1 (you can do it without loosing data, what would be quite bad :)

Beside the nice howto's on the net, here "the dark side":
- the server had LILO -- so switching that to Grub for more-easyness.
(on debian-etch, the update-grub creates a WRONG file (pointing allways to (hd0,0) as root-device, even if this is in my case the swap device...
- e2fsck.. the disk wasn't checked for very long time -- so it will at next boot. (which is maybe bad, when you want to check if the fiddle-with-the grub to boot from MD device will work; it may take hours? so doing it once before you touch the working system )
- for copy of sda to the created md device (a raid 1 with a non-filled disk) rsync is cool. it will give you several trys. (doing first the "big" data-move, then only deltas are transfered.)
I had to exclude some caching-directory to avoid that several-million-few-byte files. (they will regenerate themself later)
- grub.. its-a-trap. fiddling with the (hd1,0) (this is /dev/sdb1 ) is always a pain.


helpful for chroot:
===================
mount -o bind /dev /mnt/root/dev
mount -t proc none /mnt/root/proc
mount -t sysfs none /mnt/root/sysfs

here quick story:
got a PC with old system on-disk, but new motherboard -- with a not out-of-the-box network interface.
The Chip onboard was too new, not even in the (atm current) 2.26.30 kernel...
Good, the Vendor has some module to compile available.

now... I need to bring some stuff to the disk. (e.g. gcc, make, kernel-headers, ... )
I learned in this journey a lot of nice possibilities:

To note, the box had no space for a other card, not I had a usb-wlan stick around...

* "memstick" ( the usb thumb drive thing -- this includes a lot of walking ) -- fast (apt knows the --print-uri option, so you can wget them easy... but the dependencies makes you cry)
* running network over serial cable. ( slattatch is your friend ) -- very slow, did set it for 115k baud...
* network over blue-tooth (it worked one day in NAP mode, next day I had no luck to get it work again) -- should make up to 80k/s

This was the bad card:
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)

The driver to build is r8168.ko

The GRML (grml.org) live-CD had the driver in -- so a boot/mount/chroot trick makes it, too. (very handy !)

hi,

ok, a quite old problem: you run out of disk-space... ( at least for me - give me a 100tb disk, and I will have the same issue again soon :)
On my debian lenny my root-partition is quite full ( > 95% ). not nice, as this can quick lead to problems.
So.. who uses all the disk ? here some tips:

1) get rid of some temp-files.
apt-get clean # will delete the cached .deb packages;

2) by directory: a
du -x --max-depth=1 /
will show you in root-dir (/) the size of each directory up-to-level 1 (means e.g , /home, but not /home/user ), excluding other mounted file-systems. ( like /tmp or /var )
you get a bit a feeling in which directory the huge files are... ( so if /root shows up, you knwo whos guilty :)

3) this (quick & dirty hacked) script:
it will write-out the installed packages, re-read them and search in dpkg.status for the installed-size info.
you will e.g. see that "laptop-detect" just uses 20k, but kde-libs-data 29796 k.
so you might find that piece of software you do not use, but steals lot of disk-space... (this works of course just, if you install SW the "clean" .deb way)

------------&<-------------------
#!/bin/bash

function print_installed_size
{
pack=$1
si=echo grep "Package: $pack$" -n5 /var/backups/dpkg.status.0 | grep "Installed-Size"
print $si
}


dpkg -l | awk '{print $2}' > /tmp/installed_packages.txt

cat /tmp/installed_packages.txt | while read p
do
echo $p
print_installed_size $p
done
------------&<-------------------

sometimes you need to get rid of just _some_ rules, without touching the rest.
( as you are in a running environment, and dont want to interrupt others )

example:

you run a NAT
(this is a manualy modified stupid example to not leak my ip's - so it might contains errors :)

# map network

for i in 10 11 12
do
iptables -t nat -I PREROUTING -d 10.0.0.$i -j DNAT --to 192.168.1.$i
iptables -t nat -I POSTROUTING -s 192.168.1.$i -j SNAT --to 10.0.0.$i
done


listing what you have:

iptables -L --line-numbers -t nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DNAT 0 -- anywhere 192.168.1.10
2 DNAT 0 -- anywhere 192.168.1.11
3 DNAT 0 -- anywhere 192.168.1.12

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 SNAT 0 -- 192.168.1.10 anywhere to:10.0.0.10
2 SNAT 0 -- 192.168.1.11 anywhere to:10.0.0.11
3 SNAT 0 -- 192.168.1.12 anywhere to:10.0.0.12

you see the numbers in front ? now you can delete based on them
e.g.

iptables -t nat -D POSTROUTING 1

will delete the
1 SNAT 0 -- 192.168.1.10 anywhere to:10.0.0.10
rule, without touching the rest.
(of course, you will want to delete the PREROUTING, too. )

had just a ugly fight with GRUB.
(the server stands remote, so i cant watch boot-process, just let it remote-reset and let boot into a rescue-mode)

Had installed a actual ETCH, but very old kernel, lilo.

So i decided to switch from lilo to grub and boot a actual kernel.

The most annoying thing was, that i can install grub ( grub-install ) in a broken way. (so it will not boot)
Also, the fist 512byte of the disk (=MBR) do contain still a "LILO" string.

As my clam-daemon from my exim-mailserver did die the 2nd time ( for days -- stop accepting emails ) I built in a small check-script.

#!/bin/sh
# use this e.g. in cron.hourly

logger -t clamAlive "checking for clamd..."
estr="malware acl condition: clamd: unable to connect to UNIX socket /var/run/clamav/clamd.ctl"
logfile="/var/log/exim4/mainlog"
daemon="/etc/init.d/clamav-daemon"
misscount=$(tail -n 50 $logfile | grep "$estr" | wc -l);

# if we get 5 messages ( cant connect to clamd ), try to restart it...
if [[ $misscount -gt 5 ]]
then
logger -t clamAlive "clamscan seems to be dead. (error) try to restart..."
sh -c "$daemon stop"
sh -c "$daemon start"
mailx -s "clamd (your-domain-goes-here) did die? restarted." your@email.here </dev/null else logger -t clamAlive "clamscan seems to be active. (ok)" fi

i had to re-generate ssl-keys for my apache2 ( debian.. ) ( btw, there is a nice test on heise.de )

found no _easy_ howto on the net ( somehow, they did not work for me.. .)

anyway, doing it the apache.org way ( http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keyscerts )

create a new key:
openssl genrsa -out server.key 1024 # no -des3, as my apache should start without password :)

create a CSR:
openssl req -new -key server.key -out server.csr # -config openssl.conf if available to answer all the questions.

open cacert.org, copy-paste it there.

get the CRT back, save it.

in apachec.conf ( or vhost )

SSLCertificateFile /path/to/this/server.crt
SSLCertificateKeyFile /path/to/this/server.key

thats it.

Ok, there was this debakle with non-random ssh-keys generated on debian-like boxes.

It was a bunch of work, but I fixed now everything.
Some notes:
- there tool "ssh-vulnkey -a" is great. helped me a lot.
- funny, I detected that some of my etch-boxes are quite old -- the keys was generated under sarge, and therefore non-vulnerable; anyway, to be save I regenerated them.
- now, when all keys changed, you get lots of warnings from your known_hosts file. theres the "ssh-keygen -R host.name" trick to delete out this special one key.

In my eyes, debian has not lost his face with that error.
- the patch introduced once was discussed with the openssh team...
- they did a good work in publishing the problem. providing tools, wiki-pages, ....
- saying "ok, we have a massive problem, but here's how to fix that" has my full respect. some others should take this as a good example ( ok, don't believe MS will now take back there vista and ask for excuse ;-)

Some more learnings from vmware-server.

- vmware does not support disk-paritions being lvm-volumes. :-(
there is some dirty hack ( in the end its switching some bits in the binary.. ) to disable this limit.
- I run vmware that way:
each users gets a lvm-volume ( named by user ), formated. Here he can place his disk-files. this has several pro's.
1) user has his fixed disk-space, ( some sort of quota )
2) his-is-his.
3) you can do fs-check per vm
4) you can use lvm-backup-snapshoot per vm ( not try'd so far )
5) in the end I have a user A with vmware-process under user A in a partition A chown'ed to user A.
also using some shr volume to share iso-images.
- vmnet1 does only support "private" network-addresses ( like the 192.168.x.x ), not regular ones.

mpopenid is a openid-based htaccess replacement.
Instead of sending a 401-auth-requested with user/pass, you can log in with your openid-id.
of corse this does not just authentication ( = you have a openid ) but also autorization ( limit access to some id´s only )
So - this provides a nice single-sign-on.


There are some footnotes to consider.

1)
you need apache2 and mod_python
be sure this setup is working.

you can do e.g. by adding this to apache2-config:

<Location /mpinfo>
   SetHandler mod_python
   PythonInterpreter main_interpreter
   PythonHandler mod_python.testhandler
</Location>

2)
get the neccessary libraries:
- python-openid-1.x ( yes, use the 1.x version, not the 2.x :-(   )
- pyhton-yadis lib
- urljr fetchter.
all are avaliable here http://www.openidenabled.com/python-openid/

in a python-shell a "import openid" should now work.

3)
get the mpopenid lib.
available here: http://j3h.janrain.com/mpopenid/

4)
protect a site
<directory /web/www.example.com/html/protected>
  PythonAccessHandler mpopenid::protect
  PythonOption authorized-users "http://justme.openidprovider.com/"
  PythonOption action-path "/openid"
</directory>

<Location "/openid">
  SetHandler mod_python
  PythonOption action-path "/openid"
  PythonOption store-type file
  PythonOption store-directory /tmp
  PythonHandler mpopenid::openid
</Location>


for all pyhton-install's its quite easy:
uncompress, untar, ( tar -xzf xxx is your friend )
then go the the directory and install using
python setup.py install


some random thougths:
- the mpopenid seems to be quite old. still supported ?
- installing python things works nice, but what about updating / upgrading / un-installing ?
- you can e.g. using your livejournal openid  ( or setup your own server )

before, I had tested the libapache2-mod-auth-openid - but this does just authentication no authorization. ( so, everyone with a openid can access the page.. )

test - News funktionieren.. :)

as I've read, installing "vmware tools" can give you also some performance boost :
- the vmnet network-driver should bypass some levels of abstraction - more MB/sec at lower latacy [1] [2]
- if you use a graphic desktop, the video/mouse driver should run nice; also drag&drop might work.
- some performance-killers can also be: irq [1], chipset [3]

1] http://www.tuxyturvy.com/blog/index.php?/archives/37-Troubleshooting-VMware-ESX-network-performance.html
2] http://communities.vmware.com/thread/108982
3] http://forums.fedoraforum.org/forum/showthread.php?t=105185&highlight=nvnet

you can run a linux inside vmware on a linux-host. quite easy. ( talking about the free vmware-server )
But: if you start talk about performance, things are not so easy:
- a vm-client has no phys access to most of it´s components: disk, net, cpu, ...
- a vm-server has to handle tons of request from each system... being fair and fast is difficult.

so here some of my learnings:
- run NTP inside each vmware client. ( else you will end up in wrong system-time )
- kernel disk-io scheduler: deadline for host, noop for client; ( so host can do a fair scheduling, as client does not need to reorder things for it's virtual disk )
- at least my boxes do have limited memmory. (even if memmory it is cheap, it's a hosted box) so strip-down all your clients th their real needs. ( do you need 20 running apache's ? does mysql really need its 100mb memmory for these few requests ? )
- if a client runs out-of memmory, it starts swapping... hello disk...
- as from saying, the new ( > 2.6.21 ? ) tickless kernels should allow you do some power-saving on CPU.
- choosing 32 or 64 for host ? ( i use 64bit, but have no realy numbers to compare. )
- disks... are bad. they do break. so think on raid ( 1, 5 ... not 0 :)
- do backups. eigther whole disk-images or backing up the client.
- lvm (on host) helps you correct worng estimated disk-sizes. also a suspend-vm - lvm-snapshoot - start vm gives you a nice chance to backup data consistent and quick.
- monitor your box. use SMART for disks ( sometimes the do tell you they will die ). use sensors for reading cpu-, MB-, fan-, disk- temperature. ( hot disks will die earlier )

Version 1.1 is ready for download.
It fixes the 2.4. python self.path[0] problem.

tellus:~# hdparm -tT /dev/onetb/video

/dev/onetb/video:
Timing cached reads: 1696 MB in 2.00 seconds = 846.01 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
Timing buffered disk reads: 246 MB in 3.02 seconds = 81.55 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

so.. 85MB/sec read .. not bad. ( sinlge disk ca. 60MB )

Using BackupPc and storing to a external Harddrive. ( usb )
But, it seems not to be garunteed that its always mounted...

So, this helper-scritp, runned as "DumpPreUserCmd" does the job for me.

backup_box:/usr/share/backuppc/bin$ more is_disk_mounted
#!/bin/bash

grep "backup_platte" /etc/mtab # disk is mounted to /mnt/backup_platte ( --> fstab )
if [[ $? -ne 0 ]]
then
echo "Externe Platte mounten..."
/bin/mount /mnt/backup_platte
else
echo "Platte ist bereits gemountet."
exit 0
fi

grep "backup_platte" /etc/mtab
if [[ $? -ne 0 ]]
then
echo "...fehlgeschlagen !"
else
echo "... ok!"
fi

the problem is well know:
- a computer was infected and is now part of a BOT net.
- the victim (e.g. mail-server getting tonns of spam ) does know the IP - but it's difficult to react.

idea:
provide a standard communicaiton-interface:
- a collection-client on victim-side, collecting infos about who-when-what.
- pack to per-ISP packets
- administrator sign [1] the packet, transfer in a standard-format to the ISP. [2]
- ISP can find out, who had when the IP and inform computer-owner by email about possible virus/trojan activities. [3]

1] digitally sign is a must, to build some trust.
2] e.g. a soap-interface. ( something XML based... :)
3] true: there are 1000's of ISP's who do not care to much about their customers, but only see them as money-cows; and there 1000's of computer-users have no feeling for security. ( runing windows without updates and virus-scanner is like drive drunken )

strange, python seems to have changed in version 2.4:
Any of the sys.path[0] point to anywhere, but not the current working-directory.

see also e.g. http://effbot.org/pyref/sys.path.htm

"As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter."

Not, it isn't - at least not on my box.

To fix it, just search/replace any of the "sys.path[0]" by "." ( so path will become e.g. "./langs.xml" )
A fixed version ( 1.01 ? ) will be posted later.

make a existing lvm device bigger...

ugh - we run out of space !

# df
/dev/mapper/onetb-vmware 20158332 19089688 44644 100% /mnt/onetb/vmware

# lvextend -L+20G /dev/onetb/vmware
  Extending logical volume vmware to 39,53 GB
  Logical volume vmware successfully resized

so we have now "more space on the disk"

next, we need to let the ext3 know about this...

# umount /mnt/onetb/vmware
( e2fsck on mouted devices is bad... - but it will let you know ;-)

#:/# e2fsck -f /dev/onetb/vmware
e2fsck 1.37 (21-Mar-2005)
Durchgang 1: Prüfe Inodes, Blocks, und Größen
Durchgang 2: Prüfe Verzeichnis Struktur
Durchgang 3: Prüfe Verzeichnis Verknüpfungen
Durchgang 4: Überprüfe die Referenzzähler
Durchgang 5: Überprüfe Gruppe Zusammenfassung

/dev/onetb/vmware: ***** DATEISYSTEM WURDE VERÄNDERT *****
/dev/onetb/vmware: 71/2562240 Dateien (19.7% nicht zusammenhängend), 4852839/5120000 Blöcke

now do the resize...

#:/ resize2fs /dev/onetb/vmware
resize2fs 1.37 (21-Mar-2005)
Resizing the filesystem on /dev/onetb/vmware to 10362880 (4k) blocks.
Das Dateisystem auf /dev/onetb/vmware ist nun 10362880 Blöcke groß.

hehe, problem solved :-)
# df
/dev/mapper/onetb-vmware 40802112 19089688 19639848 50% /mnt/onetb/vmware

I had the problem, that truecrypt won't install under debian/sarge.
Got a error like ... md.h not found.
after a bit searching I found, that only kernel-headers was linked against /usr/src/linux
linux -> kernel-headers-2.6.8-2-k7/

so, get the kernel-source, and re-link. --> tataa.. it works now.