Hetzner - DokuWiki

Hetzner Rescue-System/en


What is the Hetzner Rescue-System?

The rescue system is a minimal Linux system, that can be started on demand from a Hetzner-Bootserver on the network. It is running completely in memory without using the local disc. You have full acess to all disc and can for example repair file systems or install a new operating system. The rescue system is based on Debian Linux and contains many useful tools that are required to diagnose and repair a damaged system. It is also easliy possible to install any Debian packages.

How Do I Start the Rescue System?

The Rescue system is activated from the Hetzner Robot. After choosing the desired server from the server overview (Menu "Server" -> click on the desired server) go to the tab "Rescue". Here you can activate the rescue system for every one of your servers. After activating the rescue system, an automatically generated password is displayed. Please make a note of it or remember it as this is the root password of the rescue system.

Depending on the architecture (64 bit or 32 bit) installed system, the corresponding radio button should be selected.

Finally the system must be rebooted. The server will then boot the rescue system from the network instead from hard disc.

A reboot can be done by:

  • on the console with the command "shutdown -r NOW" or a plain "reboot"
  • over the "Reset" interface in the Hetzner Robot
  • or by requesting a manual reboot by a datacenter technician from Hetzner (only during business hours. Currently: Monday through Friday between 6:30am and 10:45pm and Saturday between 10am and 5pm)

After the reboot you can connect via SSH to the normal IP address of the server and login as root with the previously noted password.

If you are happy now that you have a console again, then you probably don't need to read any further. For everybody else here are a few hints:

How to I get to the data of my server?

Run the following commands to mount the hard drive

(this example applies to the EQ series:
Warning! Older servers without software RAID, which are also still using the old IDE drivers, must use /dev/hda2 instead of /dev/md2!)

  cd /mnt
  mkdir rescue
  mount -o ro /dev/md2 /mnt/rescue

The disc is now mounted in "read only" mode. If you wish to write to the disc, it must be mounted as follows:

  mount /dev/md2 /mnt/rescue

If you want to do something more than just to get to your data, like changing the kernel or something similar, additional (virtual) filesystems should be mounted:

  mount --bind /dev /mnt/rescue/dev/
  mount --bind /proc /mnt/rescue/proc/
  mount --bind /sys /mnt/rescue/sys/
  # or shorter with a script
  chroot-prepare /mnt/rescue

now it is possible to enter the installed system with

  chroot /mnt/rescue /bin/bash

How do I run a filesystem check?

If you are seeing error messages in syslog, your file system becomes readonly or if you just suspect a problem with the filesystem, you can check the filesystem. Unfortunately this should to be done when the filesystem is not mounted. To start a filesystem check run the following command (example EQ series):

  fsck -C0 /dev/md2

The -C0 displays a progress bar. For every error found you will be asked whether it should be repaired or not. If all errors should be corrected automatically, the -y option can be used:

  fsck -C0 -y /dev/md2

To be on the very safe side, the fsck can be run with the explicit file system type instead of autodection, e.g.:

  fsck.ext3 /dev/md2
  fsck.ext2 /dev/md2

Note: If fsck stops with "exited signal 11", it may help to use the fsck version of the installed system. To do that, the root partition must be mounted and all files required for fsck have to be copied into a directory of the rescue system. Do not forget to unmount the partition with umount before running the filesystem check!

The fsck versions for the different filesystems must be called explicitly inside the new directory. Otherwise the version of the rescue system will be run.

  ./fsck.ext3 /dev/md2

How do I re-install my server from the rescue system?

Hetzner offes a comfortable, menu driven script called installimage. For further information please refer to: Installing OS Images

How do I copy images of the partitions of my server encrypted to another server?

This short tutorial describes how partitions can by copied as image to another server with the rescue system, sshfs and partimage.

To mount a foreign directory on a server using ssh, you need to install sshfs in the rescue system and load the fuse module.

apt-get update
apt-get install sshfs
modprobe fuse

Now a directory from a remote server can be mounted encrypted.

sshfs  username@remote.host:/directory/on/remote/server/  /mnt

One way to backup partitions is partimage which is already installed in the rescue system. The program itself is not very complicated and most things should be straightforward. At the top choose the partition to be saved, in the middle type in the name of the image file and its path.

Alternatives to sshfs are NFS or SMB/CIFS mounts. But since neither of them are very common on a public Linux server, sshfs is usually the best choice. Also because it is encrypted.

Instead of using partimage to copy single partitions, the whole hard drive can be copied using dd. However, dd copies blockwise. If it encounters an error, the program terminates. (Should this be the case, you can use dd_rescue). You should also be very careful to use the right device names for if= and of=, otherwise you might accidently erase all you data. The main disadvantage of dd is that it (unnecessarily) also copies empty parts of the hard disc.

I forgot my root password. What do I do now?

With the help of the rescue system you can set a new root password without knowing the previous one.

First mount the installed system:

 cd /mnt
 mkdir rescue
 mount /dev/md2 /mnt/rescue

Then change root into the system

 chroot /mnt/rescue

and run


where you can enter a new root password. After typing 'exit' you can reboot the server.

How do I mount a software RAID in the rescue system?

The command

cat /proc/mdstat

shows all available software RAID arrays. Normally the rescue system assembles all autodected arrays. Should your array not be automatically started, you can manually do so with the following command:

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1

Now you can mount the just created MD device with

mount /dev/md0 /mnt

Other rescue systems

In addition to the Linux-based rescue system, there are also rescue systems for the following operating systems:

© 2020. Hetzner Online GmbH. Alle Rechte vorbehalten.