I love infomercial gifs

In a recent 5 minute lightning talk on docker, I got a little carried away with the gifs. So good.



docker: devicemapper fix for “device or resource busy” (EBUSY)

This article is intended for folks familiar with docker and looking to fix particular issues encountered when using devicemapper storage/graph driver.

While this is issue not exclusive to devicemapper, the mechanics currently involved in this driver cause it to be affected by this.

A couple of the more commons issues seen when using the ‘devicemapper’ storage driver is when trying to stop and/or remove a contianer.
In the docker daemon logs, you may see output like:

[error] deviceset.go:792 Warning: error waiting for device ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef to close: Timeout while waiting for device ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef to close

or more likely:

Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy
[8ad069f7] -job rm(ac05cffda663) = ERR (1)
[error] server.go:1207 Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy
[error] server.go:110 HTTP Error: statusCode=500 Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy

What’s happening behind the scenes is that devicemapper has established a new thin snapshot device to mount then container on.
Sometime during the life of that container another PID on the host, unrelated to docker, has likely started and unshared some namespaces from the root namespace, namely the mount namespace (CLONE_NEWNS). In the mounts referenced in this unshared host PID, it includes the thin snapshot device and its mount for the container runtime.

When the container goes to stop and unmount, while it may unmount the device from the root namespace, that umount does not propogate to the unshared host PID.
When the container is removed, devicemapper attempts to remove the thin snapshot device, but since the unshared host PID includes a reference to the device in its mountinfo, the kernel sees the device as still busy (EBUSY). Despite the fact the mounts of the root mount namespace may no longer show this device and mount.

1) start the docker daemon (with debugging and forced devicemapper): `sudo docker -d -D -g devicemapper`
2) start a container: `sudo docker run -it busybox top`
3) run a pid with unshared mount namespace:
3.a) compile a simple C application:
3.b) use the unshare(1) utility: `sudo unshare -m top`
4) stop or kill the container from step #2
5) watch the docker daemon logs

If you kill/quit the unshared application from #3 while the docker daemon is attempting to remove the container, then the daemon can clean up the container nicely. Otherwise there is cruft left around (in /var/lib/docker) and the daemon will not have record of the container to be removed.

Tooling to visualize this inheritance does not really exist yet, so deriving the culprit can take a little effort.
Sometimes the `perf` tool can quickly point out the culprit. Something like:

perf probe -a clone_mnt
perf record -g -e probe:clone_mnt -aR docker -d
perf report

Another option, is that while the container is running, get the mountinfo of the container from only the docker daemon’s pid:

> sudo grep $(docker ps -q | head -1) /proc/$(cat /var/run/
173 169 253:5 / /var/lib/docker/devicemapper/mnt/7c62e78ca18f88e152debfb0b40847c1486bcef14d40300154bf0c9e9800d824 rw,relatime - ext4 /dev/mapper/docker-253:2-4980739-7c62e78ca18f88e152debfb0b40847c1486bcef14d40300154bf0c9e9800d824 rw,seclabel,discard,stripe=16,data=ordered

Kill/stop the container (`docker kill ...` or just ‘q’ out of `top`).
Now that the container and device should be unmounted everywhere, lets grep for any PIDs still holding a reference:

sudo grep -l 7c62e78ca1 /proc/*/mountinfo

We have our culprit. Find out the command:

ps -f 5132
root 5132 5131 0 17:47 pts/6 S+ 0:01 top

This is the unshared PID we started earlier.

Notice also, we can determine for sure they are on different mount namespaces by checking the following:

sudo ls -l /proc/$(cat /var/run/ /proc/5132/ns/mnt
lrwxrwxrwx. 1 root root 0 Nov 4 18:13 /proc/10388/ns/mnt -> mnt:[4026532737]
lrwxrwxrwx. 1 root root 0 Nov 4 18:13 /proc/5132/ns/mnt -> mnt:[4026531840]

Notice they are referencing different numbers here.

Now to cleanup, we can stop/kill this unshared PID, and `docker rm ...` our test container.

The solution to this issue is to have the docker docker itself run in an unshared mount namespace. Unfortunately, due to the threading model of the golang runtime, this can not be encapsulated inside the docker daemon itself, but will have to be done for the invocation of the docker daemon.

The two ways to go about this is either with the unshare(1) utility or editing the systemd unit file for the docker.server (where ever this applies).

For systemd unit file (likely /usr/lib/systemd/system/docker.service, but may also be /etc/systemd/system/docker.service), include “MountFlags=private” in the [service] section. e.g.:

Description=Docker Application Container Engine
Documentation= docker.socket

ExecStart=/usr/bin/docker -d -H fd:// -H unix://var/run/docker.sock


For the unshare(1) approach, you’ll have to find and edit the init script for your system. But the layout is as follows (and covers manually calling docker as well):

sudo unshare -m docker -- -d

The important piece being the “-m” flag for cloning the mount namespace, and the “--” for the separation before arguments to the docker executable itself.

What this will accomplish is at the launch of the docker daemon, it will get its own private mount namespace that is a clone of the root namespace at that point. The docker daemon is free to create devices, mount them, unmount them and remove the device, because none of that mount information will be in the root mount namespace nor subject to being cloned into other PID’s mount namespace.


Tastes in Languages: an internal dialogue

This has been a point of contention for me, ever since I have had a desire to learn a “language” other than shell scripting. Which language would be most practical, valuable, powerful, reach the most receptive audience, designed with the ideals you share, what pays the bills, etc. Some, or all, of these aspects can conflict at any given point.

As for being practical or powerful, these measurements are based on your use of the language, which vary more than there are languages, so it is a continually refining target. As I continue to learn more languages, and deepen my knowledge of others, it is evident that no one language can be the end-all. Learning their benefits and drawbacks is a large part of efficient implementation.

As for valuable, this can be in a business sense, or marketability since. For me, this is the first real point of indecision. Fortunately AND unfortunately, this measurement has a callback to another measurement, “What are the ideals and motives of the language’s design?”. Business sense can be a fuzzy option sometimes. If you are a business unit that is exclusively a Java shop, it does not mean that all glue and helpers must also be java. Inversely, just because Perl is flexible and powerful, does not mean it needs to be built into a megalithic solution. The marketability of a language though, can relate more to the individual developer.

The evolution of all the available languages comes due to varying factors and motivation, and that motivation assists greatly in the type of community and reputation that a language accrues. These are the ideals that a language is evolved with. This is the perception that can be reflected back on a developer. Like how PHP was formed by a loose group of code hackers, perl was an overly flexible parsing and report language, ruby was started by a single humble person seeking to please others. Thus is the seed that grows the community. As soon as you state what language you write in, the other person immediately conjures stereotypes to cast over you. Regardless of good or bad, it is another expression of the eight mundane concerns. If it is blame or disapproval that you seek to avoid, then you find yourself not associating with a broader audience. And seeking praise and approval, you turn in towards like minded folks to reaffirm your expressed position. This may work out just fine for many, but it does put you at a great chance of clinging to an us-and-them mentality, which can be poisonous for you and the community at large. I’m not going to expand on that greatly, you can find more insight on that, starting by reading on ingroups and outgroups

And for many, they may not be an advocate of the language they know best, but it is what has been taught abd/or what they are paid to write in. That is okay, because you can learn other languages, despite the stereotypes others cast on you from what you’ve written in. There are soon likely to be a multitude of .Net folks, regardless whether loyalist or indifferent, that need to branch out to new languages. Thankfully for the open source communities at large, these folks have fodder to learn from, and become involved with to broaden their own experience.

While I have languages that I am more comfortable with, than others, ultimately I am an advocate for being a language generalist. Determining the best tool for the job requires having a broad idea of the pros/cons of many options. Forcing a project into the box you are most familiar with, is not always the most efficient or effective approach.

Good Times

This year has been a nice for socializing with nerds. Attending conferences is something that can get tiresome, if the content is something you do not find interesting or stimulating. Thankfully, I have no required conferences, so I can be choosy (schedules permitting). Naturally open source and languages would top the list of places to attend.

First off, was hanging out with the KDE folks in San Francisco, CA for CampKDE – April 4,5 2011. There were a number of good talks, and a great opportunity to shake the hands of names that I have seen around, as well as meeting many new folks. The kde-promo team has a YouTube channel, that they have published all the talks and interviews to.
The talk I gave there is titled “Slackware: Quickly and Easily Manage Your KDE SC Hacking.” You can get the slides in [PDF] or [ODP], plus the videos posted on the kde-promo channel have the full talk (, as well as an interview afterwards by Wade Olson (

Next, was down to Spartanburg, SC for the SouthEast Linux Fest (SELF) – June 10-12, 2011.
This is the third year that I have attended SELF, and second time to speak, but what differentiated this year from any other speaking engagement (in the past, or distant future), was that it was a talk title “Slackware Demystified”, and none other than the founder of Slackware Linux, Patrick Volkerding was not only in attendance, but sitting on the front row! The slides from this presentation are available in [HTML] or [PDF]. Unfortunately, the videos have not been published yet. Hopefully they will actually get them published, unlike the previous two years…

Lastly … so far, was a local conference, that I did not speak at, only attended. JrubyConf – August 3-5, 2011. While I use MRI Ruby much more than JRuby, this conference was a great way to be around and hear from many brilliant folks (Like Wayne Seguin, Charles Nutter (headius), Nick Sieger, and Jim Weirich, to name a few), plus I felt that I needed to make up for missing out on RubyConf taking place in Baltimore, MD.

The KDE folks strongly encouraged making it to the DesktopSummit, which was hosted in Berling, Germany this year. While it was surely appetising to think of attending, it did not work out this time. It would have been nice to shake hands with some fellow contributors, like Eric Hameleers (alienBob). Better luck next year.

All good times, I look forward to next year, or event the rest of 2011.

Feel free to leave feedback on the talks.

Take care,

More 1337 for your Slackware-13.37 release

If you are one that is not afraid to recompile your kernel, then here is a little treat for you.

This patch applies to the Linux kernel source stored in /usr/src/linux of your Slackware Linux 13.37 install. If you do not have a ‘.config’ file present, then do

#> zcat /proc/config.gz > /usr/src/linux/.config
#> cd /usr/src/linux
#> zcat $PATH_TO_DOWNLOADS/linux- | patch --backup -p2

Assuming there are no errors, then you can proceed along with

#> make bzImage
#> cp arch/x86/boot/bzImage /boot/vmlinuz-1337-

If you are running the ‘-smp’ kernel, then append ‘-smp’ to the kernel version. To determine if you are, then run
$> uname -r

The assumption with this option, is that this new vmlinuz will be using the kernel modules provided with the package

IF you want to differentiate this kernel from the stock installed kernel, open the ‘.config’ file, search for the CONFIG_LOCALVERSION line, and set the value to something brilliant, like


This however, will require to (re)compile all the modules and the bzImage kernel. So then it may be just as easy to do

#> make tar-pkg
#> cd tar-install
#> rm -rf boot/vmlinux* lib/firmware/
#> ts=$(date +%s)
#> makepkg -l y -c y ../linux_2.6.37.6_1337-${ts}-$(uname -m)-1_mine.tgz
#> installpkg ../linux_2.6.37.6_1337-${ts}-$(uname -m)-1_mine.tgz

Explanation of events here:

  • the vmlinux-* is usually around 18mb, and primarily used when debugging.
  • the lib/firmware/ would clobber the stock kernel-firmware package. no need in doing that.
  • the ${ts} is just a timestamp of epoch time. nothing fancy.
  • makepkg will create a Slackware package with a distinguished name and version, so as to not interfere with the stock packages
  • change the ‘_mine’ in the package name, if you have your own tagging name for packages you’ve created for your own system.

In Either Event, you will need to:

  • create an initrd, with mkinitrd, if you use one
  • adjust your /etc/lilo.conf accordingly
  • re-run `lilo`
  • reboot
  • enjoy the l33t penguins

If You Want to fetch the kernel source, rather than use /usr/src/linux, download it here

Also, so that you are not suspect of malicious kernel patches. The patch adds ‘drivers/video/logo/logo_slk_clut224.ppm’, enables this logo image to be chosen with the config option of ‘CONFIG_LOGO_SLK_CLUT224’.

Take care,

fix pm-suspend in >= 2.6.35 linux kernel

For those with a newer laptop, in my case a Lenovo, the NEC USBv3 port is cool, unused and so far, a pain in the ease of my mobile life.

The hardware info (from lspci -nnvv) is:

0f:00.0 USB Controller [0c03]: NEC Corporation Device [1033:0194] (rev 03) (prog-if 30)
 Subsystem: Lenovo Device [17aa:219c]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 64 bytes
 Interrupt: pin A routed to IRQ 18
 Region 0: Memory at f2200000 (64-bit, non-prefetchable) [size=8K]
 Capabilities: [50] Power Management version 3
 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
 Address: 0000000000000000  Data: 0000
 Capabilities: [90] MSI-X: Enable- Count=8 Masked-
 Vector table: BAR=0 offset=00001000
 PBA: BAR=0 offset=00001080
 Capabilities: [a0] Express (v2) Endpoint, MSI 00
 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
 ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
 RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
 MaxPayload 128 bytes, MaxReadReq 512 bytes
 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
 LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
 ClockPM+ Surprise- LLActRep- BwNot-
 LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
 ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 DevCap2: Completion Timeout: Not Supported, TimeoutDis+
 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
 Compliance De-emphasis: -6dB
 LnkSta2: Current De-emphasis Level: -3.5dB
 Capabilities: [100] Advanced Error Reporting
 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
 AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
 Capabilities: [140] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
 Capabilities: [150] #18
 Kernel driver in use: xhci_hcd
 Kernel modules: xhci-hcd

Up until the 2.6.35 kernel, this had been using the kernel driver “xhci”, but in >= 2.6.35, it has been renamed “xhci_hcd”.

What does this have to do with anything?  The ease of my suspending to disk. This USBv3 driver is not yet playing well with the freezing/thawing process, and will give a cryptic error in /var/log/pm-suspend.log, pointing you to line 295 of /usr/lib${LIBSUFFIX}/pm-utils/pm-functions. When it gets to the point of executing `echo -n "mem" > /sys/power/state`, it returns a permission denied. Even though the contents of /sys/power/state show something like “mem disk”.

So for pm-utils, if you

  • go to /etc/pm/config.d/
  • create a file called “unload_modules“, with contents of
  • SUSPEND_MODULES="xhci xhci_hcd"
  • it’s a space delimited value, so add others if you are having issues.

TA DA. now i can resume the comfort of a quick suspend and resume!

Take care,

vb Continue reading

Getting started with compiling Wii apps in Linux

First off, setting up a development environment, and then the ability to run the games on the Wii are very different. Since my Wii was on the latest official system upgrade (v4.3), there were not many options for getting a custom game boot loader on it. Thankfully, I went with the Indiana Pwns and it was super easy and straight forward. Now I have the Homebrew Loader installed by way of using the BootMii installer, and I am free to load my own compiled projects as well.

To get the development kit up and operational it involved a bit of guess work, but thankful I can still use my GNU C utilities.

From the “devkitPPC” directory, you’ll need the appropriate download for you host type. For me, it was devkitPPC_r21-x86_64-linux.tar.bz2

It extracts to ”devkitPPC/”. You can put this where every you’d like, but I kept everything as my limited privilege user, and created a directory $HOME/bin/opt/, and put the directory there. Next you’ll need to set a couple of ENV variables that most projects look for in their Makefile’s. In my $HOME/.bashrc I added

export DEVKITPPC=$HOME/bin/opt/devkitPPC/

The manpath is for me :)

From the portlibs folder you’ll need everything with “ppc” in the name. The Wii’s proccesor is an IBM PowerPC chipset

Once you have all of these downloaded, a simple for loop will do well.

mkdir ${DEVKITPPC}/portlibs

for file in *ppc.tar.bz2 ; do tar -C ${DEVKITPPC}/portlibs/ -xjf $file ; done

Next, download libogc-src-*.tar.bz2 from, and lets get it compiled and installed (also, it depends on the ENV variables and the libraries above). The make install below will put it in $DEVKITPPC.

mkdir libogc ; tar -C libogc -xjf libogrc-src-*.tar.bz2 ; cd libogc

make clean && make && make install

Lastly, getting libfat present and respectable. get the libfat-ogc-*.tar.bz2 tar ball

tar -C ${DEVKITPPC}/portlibs/ -xjf libfat-ogc-*.tar.bz2

cp ${DEVKITPPC}/portlibs/lib/wii/libfat.a ${DEVKITPPC}/portlibs/lib/libfat.a

At this point there is enough present to begin hacking or least compiling as a tester. To validate we have a good compiling ENV, lets try out libwiigui . It is a great framework to learn/use/get started with, but for now if we use it to a validation test. At the time of writing this, the latest was So:

mkdir libwiigui ; cd libwiigui ; unzip ../

make clean && make

You should be left with two  new files (libwiigui-demo.elf and libwiigui-demo.dol).

On the SD card that I keep my HomeBrew applications on, I make a directory sd://apps/libwiigui/ and copy these two files into that directory, and rename them boot.elf and boot.dol respectively.

Now when I put the SD card in the Wii and launch the HomeBrew loader, I have a blank button (because I didn’t take the time to add icons and descriptions), and when clicked and loaded, there it the basic libwiigui framework app!


Take care,


p.s. here are a couple of additional information links.

Various VIM builds for slackware -current

In the past couple of days I have been staying unusually aware of updates and patches from the VIM project. I have uploaded several packages that have the latest patches of the 7.2 branch. Then for kicks and giggles I pulled down the 7.3a unstable build, but it is nothing released or ready.

One thing in particular, the 7.2.446 build does have --enable-rubyinterp compiled in, for supporting the Ruby interpreter. Someone had sent a link for the lusty-explorer script for VIM. While the prospects of the script look great, I was unable to give it whirl, since the stock VIM in slackware currently does not have the ruby interpreter compiled in.

Enjoy, and feel free to drop me your feedback.

Take care,



Much time spent today surrounding blogs of sorts. Adding new services for friends and customers. Upgrading to the latest wordpress 3.0, and so forth.
I figured it may be a good time to bump this blog of my own. If for now other reason, but to express how nice, simple and easy a LAMP stack is on Linux.

After attempting to play support for a family member’s OSX server. The voodoo happening that is running many of the services, is beyond my interest of discoverying why and how Apple is venturing away from being a Unix. It seems they are going the way of Microsoft. Little signs like keeping files like /etc/resolv.conf present, but symlinked off somewhere else. Furthermore, it includes a warning the most processes of the operating system do not use it. How frustrating.

Give me a Linux host.

Take care,

using the new rpm2tgz, with flags!!

In slackware-current as of 2010-02-23,  the traditional command used for very basic archive conversion of RPMs into a TAR’d.Gzip’d archive, just got a hint of flare. :)

Normal usage is typically for easily converting an RPM that you’d like to review, like a ‘src’ RPM.

But you may find yourself saying, if I could just quickly/easily install <xyz>.rpm on my system as slackware package, it would make life a bit easier. Well now there are several flags available, to be used AT YOUR OWN RISK 😉

If you execute rpm2tgz with no arguments, it will output the following

$ rpm2tgz 
/usr/bin/rpm2tgz:  Converts RPM format to standard GNU tar + GNU zip format.
            (view converted packages with "less", install and remove
            with "installpkg", "removepkg", "pkgtool", or manually
            with "tar")

Usage:      /usr/bin/rpm2tgz [OPTION] 
            (Outputs "file.tgz")

  -s    extract the install scripts to /usr/doc/$PRGNAM-$VERSION/
          for review.
  -S    extracts the install scripts to be executed on package installation
          (only pre-install and post-install scripts used)
          USE WITH CAUTION! 
  -n    name the output package using the rpm's metadata
  -r    extract what the rpm's "requires" (dependencies)
          as documention to /usr/doc/$PRGNAM-$VERSION/
  -d    attempt a wellformed slack-desc from the rpm meta data

Everything is pretty straight forward on what it does, but lets have an example. Say you want hulu-desktop, so you download a file called “huludesktop-x86_64.rpm”, and with traditional usage of rpm2tgz, you would have a resulting archive named “huludesktop-x86_64.tgz”. This is fine if you are not going to install it, but if you do plan on installing this archive, now it does not conform to the naming convention.

With the flags available, you can specify the following command

$ sudo rpm2tgz -srdn huludesktop-x86_64.rpm

and get a resulting package called “huludesktop-0.9.7-x86_64-1.tgz”.

All flags are not needed, for every situation, and particular ones should be used with care (-S), which can and may frequently wreak havoc on your system.

Take care,