zypper-1.4.2: rewritten package selection code

Zypper has undergone a major refactoring of its package selection code recently. This was mainly to enable automatic testing and to make it easier and safer to add new features.
I'll submit zypper 1.4.2 including this change to Factory soon. It seems to work fine for the most part, but if you spot something that used to work for you in 1.4.1, please let us know via bugzilla. Don't forget to attach both zypper.log and solver test case.

The new version already enables use of unified package arguments (in install, update, and remove commands) in the form of:

[+/-][repo:][type:]name[.arch][OPevr[.arch]]

± (or ~|!)                        ; install/remove modifier
repo  =                           ; repo alias, number, or name
type  = patch|pattern|product     ; if not specified 'package' isimplied
name                              ; can even be a glob
OP    = -|=|>=|<=|>|<             ; version operator
evr   = [epoch:]version[-release] ; edition (version)

E.g. $ zypper in packman:xine-ui-0.99.5cvs20091115-0.pm.1.4 (don't forget to quote the args, if they contain ?/*/</> or spaces).

More about all this (and about a few more features added in the last months) later ...

There is still some work to do to fully support this (especially don't try foo.arch :O), but the main focus now is to make sure that all that used to work before works also now. For future plans see roadmap and comments in the SolverRequester_test code. Don't hesitate to send feedback to or discuss this stuff on zypp-devel@opensuse.org or comment this blog post.

Improved Installation Summary in Zypper

And the best thing last (for now) :O) This was one of the most wanted features for a year now (bnc #389128). Here's how it currently (zypper 1.2.0) looks.

If the architecture or the vendor of an installed package is about to be changed, the old and the new arch/vendor is shown next to the package name:

The following packages are going to change vendor:
  kde4-akonadi                openSUSE -> openSUSE Build Service
  kde4-knemo                  http://packman.links2linux.de -> openSUSE Build Service
  oxygen-icon-theme           openSUSE -> openSUSE Build Service
  oxygen-icon-theme-scalable  openSUSE -> openSUSE Build Service

Other information, like version or repository is not shown by default, like it was before (i argue that it would be too much info for the default output):

The following NEW packages are going to be installed:
  dmidecode libpkcs11-helper1 libstorage libx86emu1

The following packages are going to be upgraded:
  ConsoleKit ConsoleKit-32bit ConsoleKit-x11
...
Continue? [y/n/?] (y):

But you can toggle display of such info in the 'Continue?' dialog. Enter '?' to show all the available options:

Continue? [y/n/?] (y): ?

y - Yes, accept the summary and proceed with installation/removal of packages.
n - No, cancel the operation.
v - Toggle display of package versions.
a - Toggle display of package architectures.
r - Toggle display of repositories from which the packages will be installed.
m - Toggle display of package vendor names.
d - Toggle between showing all details and as few details as possible.
g - View the summary in pager.

[y/n/?] (y):

Enter 'v' to see the versions:

[y/n/?] (y): v
The following NEW packages are going to be installed:
  dmidecode          2.10-3.4
  libpkcs11-helper1  1.07-1.7
  libstorage         2.18.16-1.1
  libx86emu1         1.1-3.1

The following packages are going to be upgraded:
  ConsoleKit        0.3.0-1.5 -> 0.3.0-1.6
  ConsoleKit-32bit  0.3.0-1.5 -> 0.3.0-1.6
  ConsoleKit-x11    0.3.0-1.5 -> 0.3.0-1.6

If the list is too long, enter 'g' to view the result in pager.

Also, the package counts are shown at the end:

1230 packages to upgrade, 2 to downgrade, 5 new, 2 to remove, 4  to change vendor.
Overall download size: 1.02 GiB. After the operation, 175.0 KiB will be freed.

If you have colors enabled the numbers will be highlighted and the ones to-be-removed and to-change-vendor are shown in red to catch your attention.

Enjoy, and let us know what you think!

todo: Add an option to show the dependency graph of the listed packages. That will answer the questions why zypper wants to install or remove a particular package.
todo: Add an option to view detailed package info.
todo: Add an option to view the changelog (esp. of the to-be-upgraded packages).

Zypper Dialogs

I was puzzled by how to make good user dialogs (or prompts) in Zypper for quite some time. My idea was that they should:

• be translatable, including the answers (or options)
• have default option, that would be chosen with a simple Enter key stroke (or automatically in non-interactive mode)
• have the default option visibly marked
• have help
• be simple (have expert options hidden from default view)

This has gradually evolved into the following:

Download (curl) error for 'http://foo.bar/beer/content':
Error code: Connection failed
Error message: Couldn't resolve host 'foo.bar'

Abort, retry, ignore? [a/r/i/?] (a): ?

a - Skip retrieval of the file and abort current operation.
r - Try to retrieve the file again.
i - Skip retrieval of the file and try to continue with the operation without the file.
u - Change current base URI and try retrieving the file again.

[a/r/i/?] (a):

And a translated version:

Ukončiť, znova, ignorovať? [k/z/i/?] (k): ?

k - Preskoč súčasný súbor a ukonči operáciu.
z - Skús stiahnuť súbor znova.
i - Preskoč súčasný súbor a pokračuj v operácii bez neho.
u - Zmeň základnú URI adresu a skús súbor stiahnuť znova.

[k/z/i/?] (k):

• the prompt options are made of a string like "a/r/i/u", which is marked for translation. Translators are advised in the source comment about the meaning of the options and that they
  should use lowercase letters to translate them (but even a longer string would work).


• The default option is chosen by the programmer, using its position (0 in the above example). On Enter, this option would be used as an answer.


• The default option is visibly shown in separate parentheses. First i thought highlighting it with some color would be nice, but then my colleagues convinced me that showing different output in color/non-color mode was not a good idea.


• Whenever there's help or more options available, a question mark (?) is shown as one of the possible options. When entered, the help would be shown, as you can see in the examples above.


• Programmer can choose how many options should be shown, the rest will be hidden. Only the most important options should be shown, the rest will be listed after entering the '?'.

It's surely not perfect, but hopefully quite usable. Debian's apt does something similar, but i'm not sure about the translations and other details.

todo: Allow to pass arguments together with the option as part of the answer. This would allow to e.g. remove particular package from the list of to-be-installed packages when seeing the summary. Or show detailed info (e.g. changelog) of a package that caught your attention in the summary before starting the installation.

(oh, did i mention that we'd be glad to receive patches implementing any of the todo's i mention in my blog posts? :O) If you like zypper and would enjoy some coding in C++, don't hesitate to contact us).

Configure Your Zypper

So i mentioned zypper.conf in my previous post? Yes, Zypper now (since 1.2.0) reads configuration file from /etc/zypp/zypper.conf and from your $HOME/.zypper.conf. It does not have too much to offer right now, appart from the colors, but the point is now we can easily add new ones there, whenever there's a need.

If there is Zypper option that you write too often, it should have it's configuration file counterpart as well, so that it saves you typing. I actually have a list of few, collected over time from users, so i'll be adding them soon. You'll always find the latest list of supported config options here, so stay tuned (and let us know if you need something that's not there yet).

So which value will be used after all, the one from the config file, or the one passed via command line? Here is the override priority order, highest to lowest.

1. command line options
2. options from $HOME/.zypper.conf
3. options from /etc/zypp/zypper.conf
4. and the lowest priority have the ones from zypp.conf, so that you can override libzypp's settings in zypper, if zypper allows it.

You can use an alternative config file via --config custom/zypper.conf global option. If you do, the other config files (item 2 and 3 above) will be ignored.

todo: Add command for manipulating the configuration files as an alternative to manual editing. The command would list available options with current values and descriptions and would allow to set new values, or restore the defaults.

Colors in Zypper

Zypper 1.2.0 is coming to Factory soon, bringing a few new features i will briefly describe in a few posts.

My last post was about colors in terminal applications, so i'll start with colors in zypper. There is new [color] section in zypper.conf (oh, i should have started with zypper.conf.. nevermind :O), where you can enable or disable colorization of the output, tell zypper whether you use dark or light terminal background, and, finally, select your own color for each kind of output, if you don't like the defaults.

By default, zypper will print progress messages from ongoing operation in white, finished stuff in grey, result in white, errors in red (surprise :O), and apart from this general stuff there are some special things highlighted, like the package counts in the installation summary. More is yet to come (but i don't want to overcolorify it, of course).



Here are the config options. Those who do not want to wait for the new package to reach Factory can get it from zypp::Head OBS repo.

So, play around, enjoy, and let me know what you think.

Colors for Terminal Apps

This has been in my plans for a long time, but i had trouble learning how to do colored output in terminals correctly. I asked people, i googled, looked at terminfo, ncurses, but still could not find a satisfactory answer. So i started to look at other non-curses terminal apps, and i stopped already at the first one: git.

#include <unistd.h>
#include <stdlib.h>
#include <string.h>

bool has_colors()
{
  if (::isatty(STDOUT_FILENO))
  {
    char *term = ::getenv("TERM");
    if (term && ::strcmp(term, "dumb"))
      return true;
  }
  return false;
}

This (translated to a C/C++ hybrid) is all that git (and now also zypper :O) does to detect the color capability of the stdout. stout must be a terminal, and the TERM environment variable must not be "dumb". The idea is that most color-capable terminals understand the ISO 6429 (former ANSI) color control sequences, and if the above check fails for your terminal, you can configure git (and soon zypper :O) to never or always use colors (or set the TERM to "dumb").

Regarding terminfo, i found it hard to learn and an overkill for just printing colored text (of course, once learned it can be used in other ways, too, e.g. to move the cursor, turn off input echoing when writing passwords, etc.). Maybe some other time.

On the other hand, ncurses are nothing for an existing terminal application which already does lots of output the usual way - you would have to rewrite all the output handling using ncurses. With ncurses it is that you write your app either with or without it from the start. But you cannot switch to it later, nor mix it with standard C I/O (printf() or C++ streams (cout <<). At least that's my impression from what i've learned...

So, zypper is now (since 1.1.0) a bit less boring, with colors :O) The idea is to print normal messages and progress in darker color, highlight important messages like results, warnings, errors, and color some special stuff, e.g. in install summary, default options in prompts, and so on. And it should be configurable, so that anybody can adapt it to his or her terminal background and text color. I'll write more about that once it's done.

Augeas for libzypp and zypper?

The need for a configuration file for zypper (plus my wish for a nice CLI for handling it) and a visit to Raphaël Pinson's presentation at FOSDEM made me look at Augeas closely.

Augeas enables you to read and edit your config files in a way which preserves comments and formatting details, allowing humans and programs coexist peacefully when it comes to editing configuration files. More than that, you can map some special structures of the file to special parts of the augeas tree, rather than using the generic INI file parser. See the zypp.conf for example (zypper.conf will have the same structure):

[main]
## Option description.
## Default: etc.
##
# disabledoption = value

## Another option description
##

another.option = value

With Augeas you can write a lens (the file contents description) which will map such file into a tree like this:

/files/etc/zypp/zypp.conf/main/1
/files/etc/zypp/zypp.conf/main/1/description[1] = "Option description."
/files/etc/zypp/zypp.conf/main/1/description[2] = "Default: etc."
/files/etc/zypp/zypp.conf/main/1/description[3]
/files/etc/zypp/zypp.conf/main/1/commented
/files/etc/zypp/zypp.conf/main/1/disabledoption = "value"
/files/etc/zypp/zypp.conf/main/2
/files/etc/zypp/zypp.conf/main/2/description[1] = "Another option description."
/files/etc/zypp/zypp.conf/main/2/description[2]
/files/etc/zypp/zypp.conf/main/2/another.option = "value"

Note the commented node. Augeas has an API for looking for the nodes, getting values from them, setting the values, and finally saving the tree back to the config file.

What does this mean for zypper? Implementing zypper conf --list is a piece of cake, as well as zypper conf --set another.option value. Commenting or uncommenting the option is a matter of adding or removing the commented node. Getting the option description for zypper conf --help some.option is easy (and maybe it is possible to map the multi-line ## description to a single 'description' node!).

Here is a draft of lens for zypper (it will need comments and further improvements). To try it out you can crete a /etc/zypp/zypper.conf file with contents like the above example, and use the augtool like this:

$ augtool -I /the/dir/with/the/zypper.aug/file
augtool> print /files/etc/zypp/zypper.conf/
/files/etc/zypp/zypper.conf
/files/etc/zypp/zypper.conf/anon
/files/etc/zypp/zypper.conf/anon/description[1] = "Configuration file for zypper"
/files/etc/zypp/zypper.conf/anon/description[2]
/files/etc/zypp/zypper.conf/anon/description[3] = "Boolean values are 0 1 yes no on off true false"
/files/etc/zypp/zypper.conf/main
/files/etc/zypp/zypper.conf/main/1
/files/etc/zypp/zypper.conf/main/1/description[1] = "Whether to use colors"
/files/etc/zypp/zypper.conf/main/1/description[2]
/files/etc/zypp/zypper.conf/main/1/description[3] = "Default value: autodetected"
/files/etc/zypp/zypper.conf/main/1/description[4]
/files/etc/zypp/zypper.conf/main/1/description[5]
/files/etc/zypp/zypper.conf/main/1/colors = "yes"
augtool>

Type help for other commands, use TAB key for completion, and check also the Augeas home page for more info.

What does this mean for libzypp? Basically the above, with respect to libzypp's files. Currently, you either edit your repos.d/*.repo, locks, credentials, and other files by hand or you leave it to libzypp's frontends, or you at least take care not to write comments in those files, because they will all get removed upon the next change done by libzypp (what we do there is to rewrite the entire files from in-memory data upon saving). This can be easily avoided using Augeas and we can add an editing API for zypp.conf. Plus the bonuses described above.

The library package (libaugeas0) would add 300 kiB of installed size to libzypp's/zypper's dependencies, (which is not little), plus an optional 200 kiB of the official lenses (augeas-lenses), if we decide to use them.

todo: Look at Augeas' error reporting. The user can of course break the structure of the file in a way that the lens can't map it to the tree anymore. If parsing of the file will fail, we need to report why, so that the user can fix it.

Zypper 1.0.0

We're closing to the release of openSUSE 11.1 and SUSE Linux Enterprise 11. Since zypper's releases are tightly tied to those of openSUSE, this is also an important milestone for zypper. Thus, the next release of zypper will have version 1.0.0. This marks more than two years of zypper's development and the outset of implementation of new nice features.

So what's next?

Several ideas and problems appeared so far. Some need to be implemented in libzypp itself, some are purely zypper's. Here is a list of the most important things for zypper 2.

• Configuration file (.zypperrc).
  
• Nice overall install progress.
• Much improved install summary (options to view version/vendor/arch changes, changelog, ...).
• More options to handle patterns (remove, install suggested, ...).
  
• Advanced media error handling with options like eject DVD drive, select DVD drive, edit failed URI, enable/disable medium specific options.
• Fixed or removed zypper shell (can it be useful enough to be worth to maintain it?)
  
• Interface to new libzypp functionality like 'download only'.
• and more...
  

First there will be some bug fixes during the beta phase of SLE 11, mainly with respect to compatibility with SLE 10 version of rug. These (and all following releses for SLE) will be versioned 1.0.x and will eventually get also to openSUSE 11.1 via online update. After that we're ready to work on zypper 2.

Stay tuned on features.opensuse.org, this TODO file, this blog and blogs of other ZYpp hackers (see my links).

Zypper Command Reference

I needed an overview of all zypper's commands and options, so i created this little script that prints all the help texts. It can be used to search for options through all commands, or to create a reference sheet for printing like this:


$ ./zypper-help-all | fold -s | a2ps -rjB --columns 3 -o file.ps


todo: fix some inconsistencies in option names (see this thread for discussion)

todo: wrap the help texts at 79th column, bug #423007 (no more need for 'fold -s' in the command above)

zypper's wiki updated

openSUSE 11.0 is out so i synced zypper's usage page with it. This changes page may be of particular interest for those already familiar with zypper. I also took a while to reorganize the main page a bit.

If there is something wrong or something you'd like to see in those pages, let me know (or just add/change it). Feedback and help is welcome.

(Writing docs is quite some work, it took me whole week to update those pages and the man page (which still has some glitches). Didn't i already tell to myself that i must make sure to update them as soon as any change happens? Well, but that's one week less for development... :O) oh...)

More options to modifyrepo

In 10.3, zypper modifyrepo (mr) only allowed you to enable/disable one repository and enable/disable autorefresh for it. In 11.0 the options have grown in number a bit. Did i mention the nice help texts?

$ zypper help mr
modifyrepo (mr) <options> <alias|#|uri>
modifyrepo (mr) <options> <--all|--remote|--local|--medium-type>

Modify properties of repositories specified by alias, number or URI
or by the '--all, --remote, --local, --medium-type' aggregate options.

Command options:
-d, --disable             Disable the repository (but don't remove it).
-e, --enable              Enable a disabled repository.
-r, --refresh             Enable auto-refresh of the repository.
-R, --no-refresh          Disable auto-refresh of the repository.
-n, --name                Set a descriptive name for the repository.
-p, --priority <1-99>     Set priority of the repository.
-k, --keep-packages       Enable RPM files caching.
-K, --no-keep-packages    Disable RPM files caching.
-a, --all                 Apply changes to all repositories.
-l, --local               Apply changes to all local repositories.
-t, --remote              Apply changes to all remote repositories.
-m, --medium-type <type>  Apply changes to repositories of specified type.

This is what you will see in zypper 0.11.3. The aggregate options (and the -k options) were added just recently by our new colleague Josef Reidinger. Those familiar with 10.3 zypper, note that the --enable-autorefresh/-a and --disable-autorefresh options are now deprecated (-a shorthand even had to be removed because of conflict with --all) in favor of --refresh and --no-refresh options. This change was introduced for the sake of consistency with other commands' options. All the options will be described in more detail in the man page soon.

Few examples

To disable autorefresh for all repositories, do:

$ zypper mr -Ra
Nothing to change for repository 'openSUSE-DVD 11.0'.
Autorefresh has been disabled for repository 'fate'.
Nothing to change for repository 'factory'.
Nothing to change for repository 'packman'.
Autorefresh has been disabled for repository 'factory-nonoss'.
Autorefresh has been disabled for repository 'factory-debug'.
Nothing to change for repository 'vbox'.
Autorefresh has been disabled for repository 'zypp:svn'.

To disable all cd/dvd repositories, do:

$ zypper mr -d -m cd -m dvd
Repository 'openSUSE-DVD 11.0' has been sucessfully disabled.

To enable RPM caching of RPM files for all remote (http/https/ftp) repositories, do:

$ zypper mr -kt
Nothing to change for repository 'factory'.
RPM files caching has been enabled for repository 'factory-debug'.
RPM files caching has been enabled for repository 'factory-nonoss'.
RPM files caching has been enabled for repository 'fate'.
Nothing to change for repository 'packman'.
RPM files caching has been enabled for repository 'vbox'.
RPM files caching has been enabled for repository 'zypp:svn'.

Of course don't forget to use zypper clean when you run out of free disk space then :O)

zypper search is back

I said i would write about zypper's XML output almost two months ago. Hm... obviously other stuff gained higher priority (i will get back to the XML output later). In the openSUSE Updater applets, the focus has shifted from the zypp back-end to PackageKit and i focused on other zypper/libzypp features and fixes. I will write about them in the following few posts.

Just recently i re-enabled all the original zypper search functionality (except --match-all which will be hopefully comming soon). Apart from the original features, there is one frequently requested change in the presenation of the search results (and thanx to the zypp and sat-solver guys it is also fast :O).

The old zypper listed all available instances of a matching package, all versions from all repositories. But people are more often interested in what the package is about than what versions are available from each respository. Hence, since 0.11.1 you will see a list of packages grouped by their names and types with a summary column instead of repository and version column:

$ zypper search browser
Reading installed packages...

S | Name                | Summary                          | Type
--+---------------------+----------------------------------+--------
i | kim-browser         | CIM Browser for CIM-XML Protocol | package
  | mysql-query-browser | A Graphical MySQL Query Shell    | package
  | tvbrowser           | digital TV guide                 | package

This output caught a few users unprepared. I hope learning that --details/-s can turn on a detailed list similar to the old one satisfied most of them:

$ zypper search -s browser
Reading installed packages...

S | Name                | Type    | Version    | Arch   | Repository
--+---------------------+---------+------------+--------+-----------
i | kim-browser         | package | 0.3-208    | x86_64 | factory
v | kim-browser         | package | 0.3-208    | i586   | factory
  | mysql-query-browser | package | 5.0r12-183 | x86_64 | factory
  | mysql-query-browser | package | 5.0r12-183 | i586   | factory
  | tvbrowser           | package | 2.6.3-22   | noarch | factory

Since zypper 0.11.2 the summaries will be abbreviated as needed to fit the width of your console screen. This is how it looks on an 80 character wide console:

$ search music
Reading installed packages...

S | Name                  | Summary                                 | Type
--+-----------------------+-----------------------------------------+-----------
  | asc-music             | Music for Advanced Strategic Command--> | package
  | libmusicbrainz        | Library That Provides Access to the M-> | srcpackage
  | libmusicbrainz-devel  | Include Files and Libraries Mandatory-> | package
  | libmusicbrainz3       | Library That Provides Access to the M-> | srcpackage
  | libmusicbrainz3-6     | Library That Provides Access to the M-> | package
  | libmusicbrainz3-devel | Library That Provides Access to the M-> | package
i | libmusicbrainz4       | Library That Provides Access to the M-> | package
  | texlive-bin-musictex  | MusiXTeX and MusicTeX                   | package
  | texlive-musictex      | MusiXTeX and MusicTeX                   | package

Of course you can use --no-abbrev/-A global option to turn the abbreviation off - the output will be messy, but you can read the whole summaries. Another way to obtain this info is to use zypper info.

...and don't forget to try zypper help search, zypper has nice help texts :O) I'll be fixing zypper update and writing a lot of documentation in the meantime.

bash completion for zypper

Version 0.10.3 of zypper will include a bash completion script written by Marek Stopka during Hack Week 2. The script uses zypper help [command] output to retrieve the list of commands and command options in order to feed it to the command line. Given the growing number of zypper's commands and options i believe this will be a pleasant help for many users.

Thanx to Marek for doing this!

Currently the script provides completion of commands and command options. This can be extended in the future to include package name completion for the install, remove and info commands, or even repository aliases for the repo handling commands and the --repo option.