Cloning openSUSE VirtualBox Machines

... or "Waiting for device /dev/disk/by-ID/... to appear." problem

A guess many people have figured this out by now, but i still know a lot who have not, so here it goes.

There are a few caveats in cloning openSUSE or SLE VirtualBox machines. One is that openSUSE uses device ID in /etc/fstab and grub's menu.lst to reference your hard disk partitions. This is cool for physical machines, since no matter what the usual device name of your hard disk is (/dev/sda, sdb, hda, ...), the device ID always stays the same. When cloning the VirtualBox hard disk image, however, the device ID changes. This consequently causes a problem when booting your new machine with cloned image.

Another one concerns the network card's udev rules, which affect the device names. These are set to use MAC address to identify the card, which also changes when creating new virtual machine (new random MAC address is generated). After booting the cloned machine with this new network card, the original one becomes eth1, and the new one is not configured; you need to configure it, e.g. using YaST. Thanx to Mišo Žugec for this tip.

To avoid these problems, after installing a fresh openSUSE or SLE which i want to use for cloning, i change its /etc/fstab and /boot/grub/menu.lst to use the usual device files, and change the network card's udev rules from MAC to BusID. So, in nutshell, the cloning process looks like this:

1. Boot the virtual machine you want to use as base for cloning.
2. Replace the /dev/disk/by-id/...-partY with /dev/sdXY in /etc/fstab and /boot/grub/menu.lst.
3. In YaST Network module, click Edit to edit the network card, open the Hardware tab, click Change in the Udev rules frame and select BusID instead of MAC address.
   
4. Shut the machine down (from within the machine itself, not using VirtualBox's power off).
5. Find the machine's .vdi hard disk file (usually under ~/.VirtualBox/ dir) and use VBoxManage clonevdi command to clone it, for example:
   (~/.VirtualBox/HardDisks)$ VBoxManage clonevdi my-perfect-oS11.2-base.vdi oS11.2-some-experiment.vdi
6. Create new machine in VirtualBox GUI and attach the cloned disk image to it when asked. You can also copy an existing machine's .xml configuration file and change the hard disk UUID, or create a script that will do this for you.

You're ready to boot your clone now. Of course, if you cloned a machine still using the device ID for hard disk, you should still be able to boot it, if you pass the right root=/dev/sdXY and resume=/dev/sdXY parameters to grub. You can also reconfigure the network devices after booting. But preparing the base is more convenient, if you need to clone it often.

For more info see:
http://srackham.wordpress.com/cloning-and-copying-virtualbox-virtual-machines/
http://forgeftp.novell.com/lfl/.html/virtualbox.html

DaysToGo - Minimalistic SuperKaramba Theme

I easily loose track of time when i have a lot of things to do on my list. I needed something to remind me how much time i have left when deciding what do to next or whether to promise to do something more. So i made this simple SuperKaramba theme.

You can download it from here.

The theme is pretty minimalistic and only contains three text items, one using a simple perl script, and an image for the background. So you can easily change it as you wish.

To install SuperKaramba and the theme on a recent openSUSE using terminal do this:

$ sudo zypper in kde4-superkaramba 'perl(Date::Parse)'
$ wget http://www.suse.de/~jkupec/superkaramba/daystogo.tar.gz
$ cp daystogo.tar.gz .kde4/share/apps/superkaramba/themes

(Did you know you could install perl modules like this? :O)

Now if you open Superkaramba, its theme selector should appear and should already contain the new theme. Opening the theme should also unpack the tarball. See the Readme file in the daystogo theme directory for further instruction on how to edit the theme (set your own text and date).

The theme file is as simple as this:

KARAMBA  W=280 H=280

<group> x=5 y=5 w=270 h=270
image x=-10 y=-10 path="img/SUSE_Head_lt.png"
text x=10 y=0 value="<edit daystogo.theme>" color=255,255,255 fontsize=24 shadow=2 bgcolor=100,100,100 font="Sans"
text x=135 y=70 w=270 align=center sensor=program program="~/.kde4/share/apps/superkaramba/themes/daystogo/scripts/daystogo.pl '31/dec/2009 23:59'" line=1 color=255,255,255 shadow=4 bgcolor=0,50,0 fontsize=120 font="Sans" interval=900000
text x=270 y=250 w=270 align=right value="days to go" color=255,255,255 shadow=2 bgcolor=50,50,50 fontsize=24 font="Sans"
</group>

The red parts are what you want to change to suit you. See this manual for description of used items and attributes. You'll find many more there, together with examples you can use to extend this theme or create your own. Looking at other existing themes is useful, too.

Last, but not least, i want to thank Troy Watson for the pretty Geeko icon i used as the background image.

Improved Installation Summary in Zypper

And the best thing last (for now) :O) This was one of the most wanted features for a year now (bnc #389128). Here's how it currently (zypper 1.2.0) looks.

If the architecture or the vendor of an installed package is about to be changed, the old and the new arch/vendor is shown next to the package name:

The following packages are going to change vendor:
  kde4-akonadi                openSUSE -> openSUSE Build Service
  kde4-knemo                  http://packman.links2linux.de -> openSUSE Build Service
  oxygen-icon-theme           openSUSE -> openSUSE Build Service
  oxygen-icon-theme-scalable  openSUSE -> openSUSE Build Service

Other information, like version or repository is not shown by default, like it was before (i argue that it would be too much info for the default output):

The following NEW packages are going to be installed:
  dmidecode libpkcs11-helper1 libstorage libx86emu1

The following packages are going to be upgraded:
  ConsoleKit ConsoleKit-32bit ConsoleKit-x11
...
Continue? [y/n/?] (y):

But you can toggle display of such info in the 'Continue?' dialog. Enter '?' to show all the available options:

Continue? [y/n/?] (y): ?

y - Yes, accept the summary and proceed with installation/removal of packages.
n - No, cancel the operation.
v - Toggle display of package versions.
a - Toggle display of package architectures.
r - Toggle display of repositories from which the packages will be installed.
m - Toggle display of package vendor names.
d - Toggle between showing all details and as few details as possible.
g - View the summary in pager.

[y/n/?] (y):

Enter 'v' to see the versions:

[y/n/?] (y): v
The following NEW packages are going to be installed:
  dmidecode          2.10-3.4
  libpkcs11-helper1  1.07-1.7
  libstorage         2.18.16-1.1
  libx86emu1         1.1-3.1

The following packages are going to be upgraded:
  ConsoleKit        0.3.0-1.5 -> 0.3.0-1.6
  ConsoleKit-32bit  0.3.0-1.5 -> 0.3.0-1.6
  ConsoleKit-x11    0.3.0-1.5 -> 0.3.0-1.6

If the list is too long, enter 'g' to view the result in pager.

Also, the package counts are shown at the end:

1230 packages to upgrade, 2 to downgrade, 5 new, 2 to remove, 4  to change vendor.
Overall download size: 1.02 GiB. After the operation, 175.0 KiB will be freed.

If you have colors enabled the numbers will be highlighted and the ones to-be-removed and to-change-vendor are shown in red to catch your attention.

Enjoy, and let us know what you think!

todo: Add an option to show the dependency graph of the listed packages. That will answer the questions why zypper wants to install or remove a particular package.
todo: Add an option to view detailed package info.
todo: Add an option to view the changelog (esp. of the to-be-upgraded packages).

Zypper Dialogs

I was puzzled by how to make good user dialogs (or prompts) in Zypper for quite some time. My idea was that they should:

• be translatable, including the answers (or options)
• have default option, that would be chosen with a simple Enter key stroke (or automatically in non-interactive mode)
• have the default option visibly marked
• have help
• be simple (have expert options hidden from default view)

This has gradually evolved into the following:

Download (curl) error for 'http://foo.bar/beer/content':
Error code: Connection failed
Error message: Couldn't resolve host 'foo.bar'

Abort, retry, ignore? [a/r/i/?] (a): ?

a - Skip retrieval of the file and abort current operation.
r - Try to retrieve the file again.
i - Skip retrieval of the file and try to continue with the operation without the file.
u - Change current base URI and try retrieving the file again.

[a/r/i/?] (a):

And a translated version:

Ukončiť, znova, ignorovať? [k/z/i/?] (k): ?

k - Preskoč súčasný súbor a ukonči operáciu.
z - Skús stiahnuť súbor znova.
i - Preskoč súčasný súbor a pokračuj v operácii bez neho.
u - Zmeň základnú URI adresu a skús súbor stiahnuť znova.

[k/z/i/?] (k):

• the prompt options are made of a string like "a/r/i/u", which is marked for translation. Translators are advised in the source comment about the meaning of the options and that they
  should use lowercase letters to translate them (but even a longer string would work).


• The default option is chosen by the programmer, using its position (0 in the above example). On Enter, this option would be used as an answer.


• The default option is visibly shown in separate parentheses. First i thought highlighting it with some color would be nice, but then my colleagues convinced me that showing different output in color/non-color mode was not a good idea.


• Whenever there's help or more options available, a question mark (?) is shown as one of the possible options. When entered, the help would be shown, as you can see in the examples above.


• Programmer can choose how many options should be shown, the rest will be hidden. Only the most important options should be shown, the rest will be listed after entering the '?'.

It's surely not perfect, but hopefully quite usable. Debian's apt does something similar, but i'm not sure about the translations and other details.

todo: Allow to pass arguments together with the option as part of the answer. This would allow to e.g. remove particular package from the list of to-be-installed packages when seeing the summary. Or show detailed info (e.g. changelog) of a package that caught your attention in the summary before starting the installation.

(oh, did i mention that we'd be glad to receive patches implementing any of the todo's i mention in my blog posts? :O) If you like zypper and would enjoy some coding in C++, don't hesitate to contact us).

Configure Your Zypper

So i mentioned zypper.conf in my previous post? Yes, Zypper now (since 1.2.0) reads configuration file from /etc/zypp/zypper.conf and from your $HOME/.zypper.conf. It does not have too much to offer right now, appart from the colors, but the point is now we can easily add new ones there, whenever there's a need.

If there is Zypper option that you write too often, it should have it's configuration file counterpart as well, so that it saves you typing. I actually have a list of few, collected over time from users, so i'll be adding them soon. You'll always find the latest list of supported config options here, so stay tuned (and let us know if you need something that's not there yet).

So which value will be used after all, the one from the config file, or the one passed via command line? Here is the override priority order, highest to lowest.

1. command line options
2. options from $HOME/.zypper.conf
3. options from /etc/zypp/zypper.conf
4. and the lowest priority have the ones from zypp.conf, so that you can override libzypp's settings in zypper, if zypper allows it.

You can use an alternative config file via --config custom/zypper.conf global option. If you do, the other config files (item 2 and 3 above) will be ignored.

todo: Add command for manipulating the configuration files as an alternative to manual editing. The command would list available options with current values and descriptions and would allow to set new values, or restore the defaults.

Colors in Zypper

Zypper 1.2.0 is coming to Factory soon, bringing a few new features i will briefly describe in a few posts.

My last post was about colors in terminal applications, so i'll start with colors in zypper. There is new [color] section in zypper.conf (oh, i should have started with zypper.conf.. nevermind :O), where you can enable or disable colorization of the output, tell zypper whether you use dark or light terminal background, and, finally, select your own color for each kind of output, if you don't like the defaults.

By default, zypper will print progress messages from ongoing operation in white, finished stuff in grey, result in white, errors in red (surprise :O), and apart from this general stuff there are some special things highlighted, like the package counts in the installation summary. More is yet to come (but i don't want to overcolorify it, of course).



Here are the config options. Those who do not want to wait for the new package to reach Factory can get it from zypp::Head OBS repo.

So, play around, enjoy, and let me know what you think.

Colors for Terminal Apps

This has been in my plans for a long time, but i had trouble learning how to do colored output in terminals correctly. I asked people, i googled, looked at terminfo, ncurses, but still could not find a satisfactory answer. So i started to look at other non-curses terminal apps, and i stopped already at the first one: git.

#include <unistd.h>
#include <stdlib.h>
#include <string.h>

bool has_colors()
{
  if (::isatty(STDOUT_FILENO))
  {
    char *term = ::getenv("TERM");
    if (term && ::strcmp(term, "dumb"))
      return true;
  }
  return false;
}

This (translated to a C/C++ hybrid) is all that git (and now also zypper :O) does to detect the color capability of the stdout. stout must be a terminal, and the TERM environment variable must not be "dumb". The idea is that most color-capable terminals understand the ISO 6429 (former ANSI) color control sequences, and if the above check fails for your terminal, you can configure git (and soon zypper :O) to never or always use colors (or set the TERM to "dumb").

Regarding terminfo, i found it hard to learn and an overkill for just printing colored text (of course, once learned it can be used in other ways, too, e.g. to move the cursor, turn off input echoing when writing passwords, etc.). Maybe some other time.

On the other hand, ncurses are nothing for an existing terminal application which already does lots of output the usual way - you would have to rewrite all the output handling using ncurses. With ncurses it is that you write your app either with or without it from the start. But you cannot switch to it later, nor mix it with standard C I/O (printf() or C++ streams (cout <<). At least that's my impression from what i've learned...

So, zypper is now (since 1.1.0) a bit less boring, with colors :O) The idea is to print normal messages and progress in darker color, highlight important messages like results, warnings, errors, and color some special stuff, e.g. in install summary, default options in prompts, and so on. And it should be configurable, so that anybody can adapt it to his or her terminal background and text color. I'll write more about that once it's done.

Augeas for libzypp and zypper?

The need for a configuration file for zypper (plus my wish for a nice CLI for handling it) and a visit to Raphaël Pinson's presentation at FOSDEM made me look at Augeas closely.

Augeas enables you to read and edit your config files in a way which preserves comments and formatting details, allowing humans and programs coexist peacefully when it comes to editing configuration files. More than that, you can map some special structures of the file to special parts of the augeas tree, rather than using the generic INI file parser. See the zypp.conf for example (zypper.conf will have the same structure):

[main]
## Option description.
## Default: etc.
##
# disabledoption = value

## Another option description
##

another.option = value

With Augeas you can write a lens (the file contents description) which will map such file into a tree like this:

/files/etc/zypp/zypp.conf/main/1
/files/etc/zypp/zypp.conf/main/1/description[1] = "Option description."
/files/etc/zypp/zypp.conf/main/1/description[2] = "Default: etc."
/files/etc/zypp/zypp.conf/main/1/description[3]
/files/etc/zypp/zypp.conf/main/1/commented
/files/etc/zypp/zypp.conf/main/1/disabledoption = "value"
/files/etc/zypp/zypp.conf/main/2
/files/etc/zypp/zypp.conf/main/2/description[1] = "Another option description."
/files/etc/zypp/zypp.conf/main/2/description[2]
/files/etc/zypp/zypp.conf/main/2/another.option = "value"

Note the commented node. Augeas has an API for looking for the nodes, getting values from them, setting the values, and finally saving the tree back to the config file.

What does this mean for zypper? Implementing zypper conf --list is a piece of cake, as well as zypper conf --set another.option value. Commenting or uncommenting the option is a matter of adding or removing the commented node. Getting the option description for zypper conf --help some.option is easy (and maybe it is possible to map the multi-line ## description to a single 'description' node!).

Here is a draft of lens for zypper (it will need comments and further improvements). To try it out you can crete a /etc/zypp/zypper.conf file with contents like the above example, and use the augtool like this:

$ augtool -I /the/dir/with/the/zypper.aug/file
augtool> print /files/etc/zypp/zypper.conf/
/files/etc/zypp/zypper.conf
/files/etc/zypp/zypper.conf/anon
/files/etc/zypp/zypper.conf/anon/description[1] = "Configuration file for zypper"
/files/etc/zypp/zypper.conf/anon/description[2]
/files/etc/zypp/zypper.conf/anon/description[3] = "Boolean values are 0 1 yes no on off true false"
/files/etc/zypp/zypper.conf/main
/files/etc/zypp/zypper.conf/main/1
/files/etc/zypp/zypper.conf/main/1/description[1] = "Whether to use colors"
/files/etc/zypp/zypper.conf/main/1/description[2]
/files/etc/zypp/zypper.conf/main/1/description[3] = "Default value: autodetected"
/files/etc/zypp/zypper.conf/main/1/description[4]
/files/etc/zypp/zypper.conf/main/1/description[5]
/files/etc/zypp/zypper.conf/main/1/colors = "yes"
augtool>

Type help for other commands, use TAB key for completion, and check also the Augeas home page for more info.

What does this mean for libzypp? Basically the above, with respect to libzypp's files. Currently, you either edit your repos.d/*.repo, locks, credentials, and other files by hand or you leave it to libzypp's frontends, or you at least take care not to write comments in those files, because they will all get removed upon the next change done by libzypp (what we do there is to rewrite the entire files from in-memory data upon saving). This can be easily avoided using Augeas and we can add an editing API for zypp.conf. Plus the bonuses described above.

The library package (libaugeas0) would add 300 kiB of installed size to libzypp's/zypper's dependencies, (which is not little), plus an optional 200 kiB of the official lenses (augeas-lenses), if we decide to use them.

todo: Look at Augeas' error reporting. The user can of course break the structure of the file in a way that the lens can't map it to the tree anymore. If parsing of the file will fail, we need to report why, so that the user can fix it.