Bash -o pipefail can have pitfails^Wpitfalls

Using `-o pipefail` is generally a good idea. However it can contain some surprises. For example the following shell script:

#!/bin/bash
set -e
set -o pipefail

output="$(cat /etc/passwd)"
for i in $(seq 100000); do
    echo "$output" | grep -q "^root" || res=$?
    if [ -n "$res" ]; then
        echo "failed: $i $res"
        exit 1
    fi
done
echo "done"

Does fail at different points in the loop. Why? Because grep is greedy and will exit as soon as it found a pattern. If the echo in the pipe did not finish writing to the pipe it will get a SIGPIPE when it tries to write the remaining data.

A better pattern is:

   grep -q -E "^root" <<< "$output"

Note that “-o pipefail” is already a bashism so adding “<<<" (which is also a bashism) does not make things worse 🙂

Building minimal binaries

As part of my snapd work I was writing testcode to ensure the seccomp confinement we build is working correctly. One of the challenges is that even the most simple program (like /bin/true) uses a lot of syscalls (check with strace!) so targeted testing is a bit difficult. So I decided to build a really minimal binary that avoids as many syscalls as possible.

After some experimentation the following code is ideal for our testing of the seccomp confinement:

// build with:  gcc -Wall -Werror syscall_runner.c -o syscall_runner -static -static-libgcc -nostartfiles -nostdlib -lc
#define _GNU_SOURCE
#include<unistd.h>
#include<sys/syscall.h>
void __syscall_error() {};
void _start() {
  // syscall under test with argument filtering, in this example "setpriority"
  syscall(SYS_setpriority, 1, 1, 0, 0, 0, 0);
  syscall(SYS_exit, 0, 0, 0, 0, 0, 0);
}

The above code only uses just two extra syscalls in addition to the one we want to test for:

$ strace ./syscall_runner 
execve("./syscall_runner", ["./syscall_runner"], [/* 67 vars */]) = 0
setpriority(PRIO_PGRP, 1, 0)            = -1 EPERM (Operation not permitted)
exit(0)                                 = ?

Fun!

my libproxy-golang

Sometimes it is necessary to write cgo based wrappers for existing libraries. Golang makes this pretty painless, writing https://github.com/mvo5/libproxy-golang was really quick and a fun Saturday morning project.

The only slight annoyance is the lack of convenience helpers around dealing with C `char**`. Fortunately there are some tricks like creating an “overlay” go-slice of `*C.char` which makes working with `char**` easier:

p := C.thing_that_returns_char**()
// go provides no pointer arithmetic, so make a new huge
// go slice of char* and iterate over that
tmpSlice := (*[1<<30]*C.char)(unsafe.Pointer(p))

It would be nice though if cgo had some better helpers for this.

Sysreq on Lenovo x250

I sometimes find myself in the situation that I need magic sysreq on my machine. Especially when running the development version of Ubuntu it is sometimes useful to do the magic “sync,umount,boot” sequence.

The new Lenovo keyboards do not have a dedicated sysreq key anymore. But after some searching and fiddling I is possible to emulate the behaviour:

Press “Fn”+”Ctrl”+”Alt”+”s” all at the same time. Then release “Fn”+”s” but keep “Ctrl”+”Alt” pressed. The keys you press next (while keeping “Ctrl”+”Alt” pressed) are now the systreq keys. So e.g. when pressing “s” next (while keeping “Ctrl”+”Alt” pressed) the system will do a emergency sync, pressing “u” next (while keeping “Ctrl”+”Alt” pressed) will do the emergency umount etc.

This should also work on any of the “new” Lenovo keyboard.

Booting the samsung 840 SSD firmware update iso from grub2

In order to apply a firmware update for my Samsung 840 SSD I had to boot an iso image with the firmware updater. My laptop does not have an optical drive so I decided to to use my grub2 and it’s loopback support.

So I added to

/etc/grub.d/40_custom

the following lines:

menuentry "firmware" {
  set isofile="/boot/ssd840.iso"
  loopback loop (hd0,1)$isofile
  legacy_kernel (loop)/isolinux/memdisk (loop)/isolinux/memdisk
  legacy_initrd (loop)/isolinux/btdsk.img
}

This works quite well. The firmware update process itself is a bit of a pain. At the end it asked to power cycle the SSD. Which is pretty much impossible because it is inside my laptop. After I continued without power-cycling the drive it warned me that applying failed. However that is a false scary message. After power off and power on of the laptop the firmware update was applied.

APT 1.1 released

After 1.5 years of work we released APT 1.1 this week! I’m very excited about this milestone.

The new 1.1 has some nice new features but it also improves a lot of stuff under the hood. With APT 1.0 we did add a lot of UI improvements, this time the focus is on the reliability of the acquire system and the library.

Some of the UI highlights include:

• apt install local-file.deb works
• apt build-dep foo.dsc works
• apt supports most of the common apt-get/apt-cache commands so you save some typing 🙂
• apt update progress reporting much more accurate
• apt-cache showsrc --only-source srcpkgname does the right thing
• The --force-yes option is split into the more fine grained --allow-{downgrades, remove-essential, change-held} options
• Documentation and help output improvements
• apt-mark supports more states
• Support for deb822 style sources.list.d files

Under the hood:

• No more “guessing” when fetching files (we did this to support old repository formats) only download stuff that is listed in the {,In}release file).
• support for by-hash index downloads (once the servers support that no more hashsum-mismatch errors because of proxies or transparent proxies)
• we support downloading additional files that are opaque for apt itself (like apt-file or appstream data)
• the acquire system is more atomic and more robust, no more issues with captive portals
• protection about a class of endless-data attacks from hostile MITM
• disallow signed repositories from ever becoming unsigned
• privilege dropping in the acquire methods
• if {,In}Release did not change, do not bother checking the other indexes (lot less HITs on the mirrors on not-modified resources)
• SRV record support
• improved policy engine
• key pinning for sources
• deprecation of some library functions
• support for IDN domains

Whats also very nice is that apt is now the exact same version on Ubuntu and Debian (no more delta between the packages)!

If you want to know more, there is nice video from David Kalnischkies Debconf 2015 talk about apt at https://summit.debconf.org/debconf15/meeting/216/this-apt-has-super-cow-powers/. Julian Andres Klode also wrote about the new apt some weeks ago here.

The (impressive) full changelog is available at http://metadata.ftp-master.debian.org/changelogs/main/a/apt/apt_1.1.3_changelog. And git has an even more detailed log if you are even more curious 🙂

Enjoy the new apt!

Passhash sha512crypt

I added sha512crypt support to the PassHash firefox extension a while ago to make attacking PassHash even more difficult. It uses the glibc 5000 rounds default. If you happen to use PassHash you should consider upgrading to this schema.

apt 1.0

APT 1.0 was released on the 1. April 2014 [0]! The first APT version was announced on the 1. April exactly 16 years ago [1].

The big news for this version is that we included a new “apt” binary that combines the most commonly used commands from apt-get and apt-cache. The commands are the same as their apt-get/apt-cache counterparts but with slightly different configuration options.

Currently the apt binary supports the following commands:

• list: which is similar to dpkg list and can be used with flags like
  --installed or --upgradable.
• search: works just like apt-cache search but sorted alphabetically.
• show: works like apt-cache show but hide some details that people are less likely to care about (like the hashes). The full record is still available via apt-cache show of course.
• update: just like the regular apt-get update with color output enabled.
• install,remove: adds progress output during the dpkg run.
• upgrade: the same as apt-get dist-upgrade –with-new-pkgs.
• full-upgrade: a more meaningful name for dist-upgrade.
• edit-sources: edit sources.list using $EDITOR.

Here is what the new progress looks like in 1.0:

You can enable/disable the install progress via:

# echo 'Dpkg::Progress-Fancy "1";' > /etc/apt/apt.conf.d/99progressbar

If you have further suggestions or bugreport about APT, get in touch and most importantly, have fun!

apt versions/release special modifiers

Recently the ansible apt module got fnmatch (shell) style wildcard support for installing packages. Aparently this broke the workflow for some users who passed a “*” via a variable to apt to get the candidate version installed.

A more descriptive way of achiving this is to use the one of the special words “candidate”, “installed”, “newest” in the version tag or in the release tag.

For example you can write:

# apt-get install ansible/newest
(or)
# apt-get install 2vcard=candidate

As in the ansible case, this can be a useful default for script that calcuclate a version and need to fallback to a default.

apt 0.9.12

The recently released apt 0.9.12 contains a bunch of good stuff, bugfixes and cleanups. But there are two new feature I particularly like.

The first is the new parameter “–with-new-pkgs” for the upgrade
command:

# apt-get upgrade --with-new-pkgs

that will install new dependencies on the upgrade but never remove
packages. A typical use-case is a stable system that gets a kernel
with a new kernel ABI package.

The second is “–show-progress” for
install/remove/upgrade/dist-upgrade which will show inline progress
when dpkg is running to indicate the global progress.

# apt-get install --show-progress tea
...
Selecting previously unselected package tea-data.
(Reading database ... 380116 files and directories currently installed.)
Unpacking tea-data (from .../tea-data_33.1.0-1_all.deb) ...
Progress: [ 10%]
Progress: [ 20%]
Progress: [ 30%]
Selecting previously unselected package tea.
Unpacking tea (from .../tea_33.1.0-1_amd64.deb) ...
Progress: [ 40%]
Progress: [ 50%]
Progress: [ 60%]
Processing triggers for doc-base ...
Processing 2 added doc-base files...
Registering documents with scrollkeeper...
...
Processing triggers for man-db ...
Setting up tea-data (33.1.0-1) ...
Progress: [ 70%]
Progress: [ 80%]
Setting up tea (33.1.0-1) ...
Progress: [ 90%]
Progress: [100%]

For the install progress, there is also a new experimental option
“Dpkg::Progress-Fancy”. It will display a persistent progress status bar in the last terminal line. This works like this:

# apt-get -o Dpkg::Progress-Fancy=true install tea

This kind of information is obviously most useful on complex operations like big installs or (release) upgrades.