Colors in Zypper

Zypper 1.2.0 is coming to Factory soon, bringing a few new features i will briefly describe in a few posts.

My last post was about colors in terminal applications, so i'll start with colors in zypper. There is new [color] section in zypper.conf (oh, i should have started with zypper.conf.. nevermind :O), where you can enable or disable colorization of the output, tell zypper whether you use dark or light terminal background, and, finally, select your own color for each kind of output, if you don't like the defaults.

By default, zypper will print progress messages from ongoing operation in white, finished stuff in grey, result in white, errors in red (surprise :O), and apart from this general stuff there are some special things highlighted, like the package counts in the installation summary. More is yet to come (but i don't want to overcolorify it, of course).



Here are the config options. Those who do not want to wait for the new package to reach Factory can get it from zypp::Head OBS repo.

So, play around, enjoy, and let me know what you think.

Colors for Terminal Apps

This has been in my plans for a long time, but i had trouble learning how to do colored output in terminals correctly. I asked people, i googled, looked at terminfo, ncurses, but still could not find a satisfactory answer. So i started to look at other non-curses terminal apps, and i stopped already at the first one: git.

#include <unistd.h>
#include <stdlib.h>
#include <string.h>

bool has_colors()
{
  if (::isatty(STDOUT_FILENO))
  {
    char *term = ::getenv("TERM");
    if (term && ::strcmp(term, "dumb"))
      return true;
  }
  return false;
}

This (translated to a C/C++ hybrid) is all that git (and now also zypper :O) does to detect the color capability of the stdout. stout must be a terminal, and the TERM environment variable must not be "dumb". The idea is that most color-capable terminals understand the ISO 6429 (former ANSI) color control sequences, and if the above check fails for your terminal, you can configure git (and soon zypper :O) to never or always use colors (or set the TERM to "dumb").

Regarding terminfo, i found it hard to learn and an overkill for just printing colored text (of course, once learned it can be used in other ways, too, e.g. to move the cursor, turn off input echoing when writing passwords, etc.). Maybe some other time.

On the other hand, ncurses are nothing for an existing terminal application which already does lots of output the usual way - you would have to rewrite all the output handling using ncurses. With ncurses it is that you write your app either with or without it from the start. But you cannot switch to it later, nor mix it with standard C I/O (printf() or C++ streams (cout <<). At least that's my impression from what i've learned...

So, zypper is now (since 1.1.0) a bit less boring, with colors :O) The idea is to print normal messages and progress in darker color, highlight important messages like results, warnings, errors, and color some special stuff, e.g. in install summary, default options in prompts, and so on. And it should be configurable, so that anybody can adapt it to his or her terminal background and text color. I'll write more about that once it's done.

Augeas for libzypp and zypper?

The need for a configuration file for zypper (plus my wish for a nice CLI for handling it) and a visit to Raphaël Pinson's presentation at FOSDEM made me look at Augeas closely.

Augeas enables you to read and edit your config files in a way which preserves comments and formatting details, allowing humans and programs coexist peacefully when it comes to editing configuration files. More than that, you can map some special structures of the file to special parts of the augeas tree, rather than using the generic INI file parser. See the zypp.conf for example (zypper.conf will have the same structure):

[main]
## Option description.
## Default: etc.
##
# disabledoption = value

## Another option description
##

another.option = value

With Augeas you can write a lens (the file contents description) which will map such file into a tree like this:

/files/etc/zypp/zypp.conf/main/1
/files/etc/zypp/zypp.conf/main/1/description[1] = "Option description."
/files/etc/zypp/zypp.conf/main/1/description[2] = "Default: etc."
/files/etc/zypp/zypp.conf/main/1/description[3]
/files/etc/zypp/zypp.conf/main/1/commented
/files/etc/zypp/zypp.conf/main/1/disabledoption = "value"
/files/etc/zypp/zypp.conf/main/2
/files/etc/zypp/zypp.conf/main/2/description[1] = "Another option description."
/files/etc/zypp/zypp.conf/main/2/description[2]
/files/etc/zypp/zypp.conf/main/2/another.option = "value"

Note the commented node. Augeas has an API for looking for the nodes, getting values from them, setting the values, and finally saving the tree back to the config file.

What does this mean for zypper? Implementing zypper conf --list is a piece of cake, as well as zypper conf --set another.option value. Commenting or uncommenting the option is a matter of adding or removing the commented node. Getting the option description for zypper conf --help some.option is easy (and maybe it is possible to map the multi-line ## description to a single 'description' node!).

Here is a draft of lens for zypper (it will need comments and further improvements). To try it out you can crete a /etc/zypp/zypper.conf file with contents like the above example, and use the augtool like this:

$ augtool -I /the/dir/with/the/zypper.aug/file
augtool> print /files/etc/zypp/zypper.conf/
/files/etc/zypp/zypper.conf
/files/etc/zypp/zypper.conf/anon
/files/etc/zypp/zypper.conf/anon/description[1] = "Configuration file for zypper"
/files/etc/zypp/zypper.conf/anon/description[2]
/files/etc/zypp/zypper.conf/anon/description[3] = "Boolean values are 0 1 yes no on off true false"
/files/etc/zypp/zypper.conf/main
/files/etc/zypp/zypper.conf/main/1
/files/etc/zypp/zypper.conf/main/1/description[1] = "Whether to use colors"
/files/etc/zypp/zypper.conf/main/1/description[2]
/files/etc/zypp/zypper.conf/main/1/description[3] = "Default value: autodetected"
/files/etc/zypp/zypper.conf/main/1/description[4]
/files/etc/zypp/zypper.conf/main/1/description[5]
/files/etc/zypp/zypper.conf/main/1/colors = "yes"
augtool>

Type help for other commands, use TAB key for completion, and check also the Augeas home page for more info.

What does this mean for libzypp? Basically the above, with respect to libzypp's files. Currently, you either edit your repos.d/*.repo, locks, credentials, and other files by hand or you leave it to libzypp's frontends, or you at least take care not to write comments in those files, because they will all get removed upon the next change done by libzypp (what we do there is to rewrite the entire files from in-memory data upon saving). This can be easily avoided using Augeas and we can add an editing API for zypp.conf. Plus the bonuses described above.

The library package (libaugeas0) would add 300 kiB of installed size to libzypp's/zypper's dependencies, (which is not little), plus an optional 200 kiB of the official lenses (augeas-lenses), if we decide to use them.

todo: Look at Augeas' error reporting. The user can of course break the structure of the file in a way that the lens can't map it to the tree anymore. If parsing of the file will fail, we need to report why, so that the user can fix it.

Gettext Variables

Recently i had trouble getting my zypper to talk in Slovak despite the correct locale set (using export LANG=sk_SK.utf8). I was so desperate after some time of searching for the cause that i even filed a bug report.

Well, it turned out that i forgot about one more variable, the LANGUAGE. This defines a language preference list and takes higher priority that LANG, or LC_* variables. Mine was set to en_US (i don't know why, though) :O(

So the fix for me was either to unset the LANGUAGE variable, or include sk in its value. Read more about this variable here.

RIS Specification Updated

I filled some missing parts in the Repository Index Service specification, mainly regarding client handling the repository index. I would consider it done already, but there are still a few things i would like to see done in the future:

1. The distro_target attribute should be turned into some standard distribution identifier to make it really useful. The CPE looks like a good candidate.
2. The spec does not say how to handle repositories of type unknown to the client. They should be ignored, but the type attribute of <repo> is currently optional so the type does not need to be known when refreshing the service. One solution could be to make the type attribute required, another could be to require that the client removes any repositories comming from a service that are of unknown type. The latter seems to be cumbersome, i would go for the former.
   
3. It might be worth to add the examples from this Duncan's post to the page (they're much better than those of mine :O).
4. The specification should not be kept solely as a wiki (should not be editable by broad public; version control is problematic, etc.). Any suggestion where it should be instead?

Feel free to drop a comment about the specification, or the concept, etc.

Some TODOs for libzypp:
todo: sync repo URI with the repoindex, if changed
todo: ignore repos with unknown type when refreshing service

Zypper 1.0.0

We're closing to the release of openSUSE 11.1 and SUSE Linux Enterprise 11. Since zypper's releases are tightly tied to those of openSUSE, this is also an important milestone for zypper. Thus, the next release of zypper will have version 1.0.0. This marks more than two years of zypper's development and the outset of implementation of new nice features.

So what's next?

Several ideas and problems appeared so far. Some need to be implemented in libzypp itself, some are purely zypper's. Here is a list of the most important things for zypper 2.

• Configuration file (.zypperrc).
  
• Nice overall install progress.
• Much improved install summary (options to view version/vendor/arch changes, changelog, ...).
• More options to handle patterns (remove, install suggested, ...).
  
• Advanced media error handling with options like eject DVD drive, select DVD drive, edit failed URI, enable/disable medium specific options.
• Fixed or removed zypper shell (can it be useful enough to be worth to maintain it?)
  
• Interface to new libzypp functionality like 'download only'.
• and more...
  

First there will be some bug fixes during the beta phase of SLE 11, mainly with respect to compatibility with SLE 10 version of rug. These (and all following releses for SLE) will be versioned 1.0.x and will eventually get also to openSUSE 11.1 via online update. After that we're ready to work on zypper 2.

Stay tuned on features.opensuse.org, this TODO file, this blog and blogs of other ZYpp hackers (see my links).

Zypper Command Reference

I needed an overview of all zypper's commands and options, so i created this little script that prints all the help texts. It can be used to search for options through all commands, or to create a reference sheet for printing like this:


$ ./zypper-help-all | fold -s | a2ps -rjB --columns 3 -o file.ps


todo: fix some inconsistencies in option names (see this thread for discussion)

todo: wrap the help texts at 79th column, bug #423007 (no more need for 'fold -s' in the command above)