CategoryProjects

docker: devicemapper fix for “device or resource busy” (EBUSY)

Audience:
This article is intended for folks familiar with docker and looking to fix particular issues encountered when using devicemapper storage/graph driver.

Overview:
While this is issue not exclusive to devicemapper, the mechanics currently involved in this driver cause it to be affected by this.

A couple of the more commons issues seen when using the ‘devicemapper’ storage driver is when trying to stop and/or remove a contianer.
In the docker daemon logs, you may see output like:

[error] deviceset.go:792 Warning: error waiting for device ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef to close: Timeout while waiting for device ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef to close

or more likely:

Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy
[8ad069f7] -job rm(ac05cffda663) = ERR (1)
[error] server.go:1207 Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy
[error] server.go:110 HTTP Error: statusCode=500 Cannot destroy container ac05cffda663: Driver devicemapper failed to remove root filesystem ac05cffda663a01cbc37879bc146fcd68d0f95b5b141f60da2b64579add1f4ef: Device is Busy

Diagnosis:
What’s happening behind the scenes is that devicemapper has established a new thin snapshot device to mount then container on.
Sometime during the life of that container another PID on the host, unrelated to docker, has likely started and unshared some namespaces from the root namespace, namely the mount namespace (CLONE_NEWNS). In the mounts referenced in this unshared host PID, it includes the thin snapshot device and its mount for the container runtime.

When the container goes to stop and unmount, while it may unmount the device from the root namespace, that umount does not propogate to the unshared host PID.
When the container is removed, devicemapper attempts to remove the thin snapshot device, but since the unshared host PID includes a reference to the device in its mountinfo, the kernel sees the device as still busy (EBUSY). Despite the fact the mounts of the root mount namespace may no longer show this device and mount.

Reproduction:
1) start the docker daemon (with debugging and forced devicemapper): `sudo docker -d -D -g devicemapper`
2) start a container: `sudo docker run -it busybox top`
3) run a pid with unshared mount namespace:
3.a) compile a simple C application: http://pastebin.com/HfSn8udJ
3.b) use the unshare(1) utility: `sudo unshare -m top`
4) stop or kill the container from step #2
5) watch the docker daemon logs

If you kill/quit the unshared application from #3 while the docker daemon is attempting to remove the container, then the daemon can clean up the container nicely. Otherwise there is cruft left around (in /var/lib/docker) and the daemon will not have record of the container to be removed.

Investigating:
Tooling to visualize this inheritance does not really exist yet, so deriving the culprit can take a little effort.
Sometimes the `perf` tool can quickly point out the culprit. Something like:

perf probe -a clone_mnt
perf record -g -e probe:clone_mnt -aR docker -d
[...]
perf report

Another option, is that while the container is running, get the mountinfo of the container from only the docker daemon’s pid:

> sudo grep $(docker ps -q | head -1) /proc/$(cat /var/run/docker.pid)/mountinfo
173 169 253:5 / /var/lib/docker/devicemapper/mnt/7c62e78ca18f88e152debfb0b40847c1486bcef14d40300154bf0c9e9800d824 rw,relatime - ext4 /dev/mapper/docker-253:2-4980739-7c62e78ca18f88e152debfb0b40847c1486bcef14d40300154bf0c9e9800d824 rw,seclabel,discard,stripe=16,data=ordered

Kill/stop the container (`docker kill ...` or just ‘q’ out of `top`).
Now that the container and device should be unmounted everywhere, lets grep for any PIDs still holding a reference:

sudo grep -l 7c62e78ca1 /proc/*/mountinfo
/proc/5132/mountinfo

We have our culprit. Find out the command:

ps -f 5132
UID PID PPID C STIME TTY STAT TIME CMD
root 5132 5131 0 17:47 pts/6 S+ 0:01 top

This is the unshared PID we started earlier.

Notice also, we can determine for sure they are on different mount namespaces by checking the following:

sudo ls -l /proc/$(cat /var/run/docker.pid)/ns/mnt /proc/5132/ns/mnt
lrwxrwxrwx. 1 root root 0 Nov 4 18:13 /proc/10388/ns/mnt -> mnt:[4026532737]
lrwxrwxrwx. 1 root root 0 Nov 4 18:13 /proc/5132/ns/mnt -> mnt:[4026531840]

Notice they are referencing different numbers here.

Now to cleanup, we can stop/kill this unshared PID, and `docker rm ...` our test container.

Solution:
The solution to this issue is to have the docker docker itself run in an unshared mount namespace. Unfortunately, due to the threading model of the golang runtime, this can not be encapsulated inside the docker daemon itself, but will have to be done for the invocation of the docker daemon.

The two ways to go about this is either with the unshare(1) utility or editing the systemd unit file for the docker.server (where ever this applies).

For systemd unit file (likely /usr/lib/systemd/system/docker.service, but may also be /etc/systemd/system/docker.service), include “MountFlags=private” in the [service] section. e.g.:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
ExecStart=/usr/bin/docker -d -H fd:// -H unix://var/run/docker.sock
MountFlags=private
LimitNOFILE=1048576
LimitNPROC=1048576

[Install]
WantedBy=multi-user.target

For the unshare(1) approach, you’ll have to find and edit the init script for your system. But the layout is as follows (and covers manually calling docker as well):

sudo unshare -m docker -- -d

The important piece being the “-m” flag for cloning the mount namespace, and the “--” for the separation before arguments to the docker executable itself.

What this will accomplish is at the launch of the docker daemon, it will get its own private mount namespace that is a clone of the root namespace at that point. The docker daemon is free to create devices, mount them, unmount them and remove the device, because none of that mount information will be in the root mount namespace nor subject to being cloned into other PID’s mount namespace.

Links:

Hacking on KDE: part 1

In a personal brainstorm of how to better facilitate folks that want to have an ideal hacking environment, for X. The primitives of the idea need to be figured out, and often nothing happens because you can determine “If the person has enough desire, and know how, they’ll figure it out,” and most of the time this is a fine explanation.

When it comes to a project collection like KDE, compiling it and fiddling around can be supremely easy without the bounds of package dependencies, but still manageable packages. Although, there are now a lot a projects in the KDE Software Collection, and they are all modular. Even within the Slackware community, alienBob has already updated the build process for his KDE SC 4.7 packages to be modular. With the migration from SVN to git, it has clarified some aspects of the build, but it has now made a segregated landscape of projects. If someone wants to check out a new project, the latest development in it, it seems like a hidden process.

Being agnostic of which Linux distribution I am running, if, as a developer, I want to be able to easily manage working from the git repositories of the projects, installing them, and possibly even packaging the artifacts, this is a intricate process. But thankfully, it is a describable process, therefore we can automate it.

Last night I began working on a project, tentatively named ‘kappy’. With the K App, in mind, and it is written in Python, so having a py in there seems suitable.
The source code is currently living at http://hashbangbash.com/kappy.git, or browser viewable at http://hashbangbash.com/git/?p=kappy.git;a=summary

Initially it is just doing XML parsing of the projects available. Next plans are:

  • making a caching ground for cloning/updating git repositories
  • building the software
  • having user definable configurations for flags and install paths
  • making recipes, of a suite of software to build
  • having a manager to package, for a respective distribution

Update: I’ve added a bug tracker for this project http://redmine.hashbangbash.com/projects/kappy

If you have feedback, feel free to send it in.

Take care,
vb

Good Times

This year has been a nice for socializing with nerds. Attending conferences is something that can get tiresome, if the content is something you do not find interesting or stimulating. Thankfully, I have no required conferences, so I can be choosy (schedules permitting). Naturally open source and languages would top the list of places to attend.

First off, was hanging out with the KDE folks in San Francisco, CA for CampKDE – April 4,5 2011. There were a number of good talks, and a great opportunity to shake the hands of names that I have seen around, as well as meeting many new folks. The kde-promo team has a YouTube channel, that they have published all the talks and interviews to. http://www.youtube.com/user/kdepromo
The talk I gave there is titled “Slackware: Quickly and Easily Manage Your KDE SC Hacking.” You can get the slides in [PDF] or [ODP], plus the videos posted on the kde-promo channel have the full talk (youtube.com/watch?v=Qs7vR3POHeo), as well as an interview afterwards by Wade Olson (youtube.com/watch?v=YIpUmPul1i4).

Next, was down to Spartanburg, SC for the SouthEast Linux Fest (SELF) – June 10-12, 2011.
This is the third year that I have attended SELF, and second time to speak, but what differentiated this year from any other speaking engagement (in the past, or distant future), was that it was a talk title “Slackware Demystified”, and none other than the founder of Slackware Linux, Patrick Volkerding was not only in attendance, but sitting on the front row! The slides from this presentation are available in [HTML] or [PDF]. Unfortunately, the videos have not been published yet. Hopefully they will actually get them published, unlike the previous two years…

Lastly … so far, was a local conference, that I did not speak at, only attended. JrubyConf – August 3-5, 2011. While I use MRI Ruby much more than JRuby, this conference was a great way to be around and hear from many brilliant folks (Like Wayne Seguin, Charles Nutter (headius), Nick Sieger, and Jim Weirich, to name a few), plus I felt that I needed to make up for missing out on RubyConf taking place in Baltimore, MD.

The KDE folks strongly encouraged making it to the DesktopSummit, which was hosted in Berling, Germany this year. While it was surely appetising to think of attending, it did not work out this time. It would have been nice to shake hands with some fellow contributors, like Eric Hameleers (alienBob). Better luck next year.

All good times, I look forward to next year, or event the rest of 2011.

Feel free to leave feedback on the talks.

Take care,
vb

More Red-Black Tree, but a little Ruby Benchmarking

After hearing a bit about the improvements in the Rubinius ruby interpreter, I decided to test a handful of the interpreters against the Red-Black tree insertion script I last posted about.

As a forward, this is completely out of idle curiosity, and with no deep intentions. The results had distinct winners and losers, and for more rich benchmark, this would need to include multiple iterations and some other aspects as well. All that to say, this is not an all encompassing benchmark.
The results are the length (in seconds) of time for the execution.

So using the rbtree.rb, that I last blogged about, using rvm, I have just run it through the following ruby interpreters:

  • ruby 1.9.1p430 (2010-08-16 revision 28998) [x86_64-linux]
  • rubinius 1.3.0dev (1.8.7 5bd938b5 xxxx-xx-xx JI) [x86_64-unknown-linux-gnu]
  • rubinius 1.2.0 (1.8.7 release 2010-12-21 JI) [x86_64-unknown-linux-gnu]
  • jruby 1.5.3 (ruby 1.8.7 patchlevel 249) (2010-09-28 7ca06d7) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_23) [amd64-java]
  • ruby 1.8.7 (2010-12-23 patchlevel 330) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.01
  • ruby 1.9.3dev (2011-02-14 trunk 30873) [x86_64-linux]

From an early look at it, the Rubinius interpreter is on to something good. For this example at least, their current release branch (1.2) is smoking, compared to anything else.
To look at the values, that make up the graph above, here is the spreadsheet

Take care,
vb

Kick off of playing with trees

As I have stated before, I enjoy the speed at which I can prototype an idea in Ruby. Even if ultimately that idea runs better on another language. For the sake of my academics, it provides a super learning playground.

Earlier, this past week, I began reading up on papers and documentation of trees. All in an effort to differentiate the various implementations of trees, including their benefits and drawbacks. Finding AVL trees, B trees, 2-3 node, 2-3-4 node and RB trees. I decided to kick off the events with a Red-Black tree. Some of the papers I found are the generally available by Google searching.

After a lot of learning, and a little massage, I have the first of a an “academic” implementation of an RB tree (http://hashbangbash.com/~vbatts/rbtree.rb). I gravitated towards this tree first because of the type searches I am needing. While the population of this slows as it grows, the searches are ridiculously fast. I’ll get around to generating some instrumentation to graph it. For now I have the run_loop() to return @results of the populating trees. The sets are [ , , , ]

The insertion graphed was an O(N) climb. For doubling the number of nodes to be inserted (of random unique numbers), the time increase was just a tad more than double. The @results for run_loop(1,000,000) was
=> [[1000000, 43.4912700653076, 24, 16], [500000, 20.5859882831573, 20, 15], [250000, 9.5372850894928, 20, 14], [125000, 4.50037479400635, 21, 13], [62500, 2.01573848724365, 16, 12], [31250, 0.941669225692749, 14, 11], [15625, 0.435366868972778, 14, 11], [7812, 0.252951860427856, 13, 10], [3906, 0.0940215587615967, 11, 9], [1953, 0.043536901473999, 13, 8], [976, 0.0200107097625732, 9, 8], [488, 0.00921034812927246, 9, 6], [244, 0.00417494773864746, 10, 5], [122, 0.00190210342407227, 6, 5], [61, 0.000821590423583984, 5, 4], [30, 0.000351667404174805, 4, 3], [15, 0.000170707702636719, 4, 2], [7, 6.03199005126953e-05, 2, 1], [3, 2.57492065429688e-05, 1, 1]]
which resulted in a time climb of


Another interesting observation was the logarithmic oscillations of the depth of the left most leg.

At any rate, enjoy and feel free to send your thoughts.
Take care,
vb

a point-and-clicky (ruby) addition, dobbage!

Some time ago, the brainstorm of a simple GUI package viewer for my Slackware desktop would be handy. While completely unnecessary, I thought it would be neat.
So what did I do? Created a project on github.com for an application that did not exist yet, and commenced to reading the source of others with similarly simple application frameworks.

While having the project created out on github was a reminder, it was not much for encouragement, so the idea lay dormant for a while. With a free moment this past weekend, I solidified several ideas and actually sought to make headway on this task. But I will say, that having the project on github for so long, has increased the rank on Google’s search results for “dobbage” so that is a plus :)

While I am crafty with many tools/languages, my weapon of choice remains ruby. For my development time, I can get an idea prototyped very quick. At which point I can decided whether it is worth it to use another language (for performance, etc.). All that to say, dobbage is written in Ruby.

Initially, I had thought that ruby-gtk2 would be the windowing toolkit to use, but then Qt and QtRuby are included in the stock Slackware installation.
I must say, there is not an abundance of applications to find, that are written in QtRuby, the bindings to access the Qt libraries is very nice and extensive. This is my first “GUI” application, so it’s all new to me. The Qt API documentation is written for C++, but once you get the hang of translating their C++ explanation to a Ruby implementation, then it’s all gravy! :)

Dobbage only has one dependency, that is not included in a stock Slackware install, and that is my slack-utils ruby library, which can be built as an installable Slackware package, or available as a rubygem. slack-utils does provide several command line tools, but more importantly, it includes a library for a Slackware Linux harness.

Right now, dobbage only shows information for packages currently installed, and for “removed” packages. Removed packages include previous iterations of upgraded packages. There is a little fancy-ness in the status bar too. Other ideas include the ability:

  • sorting the Removed packages by date removed, instead of alphabetical
  • to open an installable Slackware package (a *.t?z archive), and get a tab with information about that package.
  • to have a tab for fetching information from a Slackware repository /mirror
  • good ideas from others :)

In any event, the code is available at github.com/vbatts/dobbage and here are a few screen shots.

Take care,
vb

latest ruby packages for slackware current

Keeping on the track of various projects that have new releases out right now, Ruby has just released a couple of patch releases like 1.9.1-p429. Which has a list of bugfixes, and only a vulnerability that affects Windows users. Another release just a few days ago, is the 1.9.2 RC2 release. Which should be interesting for a couple of reasons. They are stating that it should be mostly compatible with previous versions.  This is the current problem with getting many ruby projects updated from 1.8.x to 1.9.1.  Now there may be potentially more updates and modifications needed.

Also, unlike the 1.8.x branch that stayed in /usr/lib/ruby/1.8/, once they got underway with 1.9.1 the path standard became /usr/lib/ruby/1.9.1/, so it’ll be changing with every sub-major release (so you would think). This is fine and good, and will join the ranks of the layout that other interpreters have taken. I just hope that the community cleanly grabs on to this, otherwise it’ll make for ugly clean-up. Another thing, is that despite these path changes due to version, there is no option (like there is when compiling perl) to -Dinc_version_list=''. So you can quickly and easy lose libraries that you had built for a previous version. We will have to continue pulling through this 1.9.2RC2 release, because initially there is not a lot of consistency in paths. There are some libraries built out as 1.9.1, and other paths as 1.9.2. Even the default headers have a mix of whether to be in a 1.9.1 or  1.9.2. Hmmm. I have taken steps to avoid these conflicts, but /usr/lib64/libruby.so.1.9.1 will still conflict.

The packages have landed on cardinal and are to be used at your own risk. I built the 1.9.2 version with a suffix of 192, in efforts to test having multiple versions coexist, but due to the spattering of 1.9.1 conflicts, you should `removepkg ruby` before installing the 1.9.2 package.

In other neat things about ruby-1.9.2, the Time class has been reimplemented so that the year 2038 is no longer a problem.

Available on http://cardinal.lizella.net/~vbatts/ you’ll find a ruby18 package. This package does coexist nicely with either ruby or ruby192. I had bundled this up while working with some rails projects (like radiant CMS!) that are not yet ready for ruby-1.9.x

Take care,

vb

Various VIM builds for slackware -current

In the past couple of days I have been staying unusually aware of updates and patches from the VIM project. I have uploaded several packages that have the latest patches of the 7.2 branch. Then for kicks and giggles I pulled down the 7.3a unstable build, but it is nothing released or ready.

One thing in particular, the 7.2.446 build does have --enable-rubyinterp compiled in, for supporting the Ruby interpreter. Someone had sent a link for the lusty-explorer script for VIM. While the prospects of the script look great, I was unable to give it whirl, since the stock VIM in slackware currently does not have the ruby interpreter compiled in.

Enjoy, and feel free to drop me your feedback.

Take care,

vb

using the new rpm2tgz, with flags!!

In slackware-current as of 2010-02-23,  the traditional command used for very basic archive conversion of RPMs into a TAR’d.Gzip’d archive, just got a hint of flare. :)

Normal usage is typically for easily converting an RPM that you’d like to review, like a ‘src’ RPM.

But you may find yourself saying, if I could just quickly/easily install <xyz>.rpm on my system as slackware package, it would make life a bit easier. Well now there are several flags available, to be used AT YOUR OWN RISK 😉

If you execute rpm2tgz with no arguments, it will output the following

$ rpm2tgz 
/usr/bin/rpm2tgz:  Converts RPM format to standard GNU tar + GNU zip format.
            (view converted packages with "less", install and remove
            with "installpkg", "removepkg", "pkgtool", or manually
            with "tar")

Usage:      /usr/bin/rpm2tgz [OPTION] 
            (Outputs "file.tgz")

  -s    extract the install scripts to /usr/doc/$PRGNAM-$VERSION/
          for review.
  -S    extracts the install scripts to be executed on package installation
          (only pre-install and post-install scripts used)
          USE WITH CAUTION! 
  -n    name the output package using the rpm's metadata
  -r    extract what the rpm's "requires" (dependencies)
          as documention to /usr/doc/$PRGNAM-$VERSION/
  -d    attempt a wellformed slack-desc from the rpm meta data

Everything is pretty straight forward on what it does, but lets have an example. Say you want hulu-desktop, so you download a file called “huludesktop-x86_64.rpm”, and with traditional usage of rpm2tgz, you would have a resulting archive named “huludesktop-x86_64.tgz”. This is fine if you are not going to install it, but if you do plan on installing this archive, now it does not conform to the naming convention.

With the flags available, you can specify the following command

$ sudo rpm2tgz -srdn huludesktop-x86_64.rpm

and get a resulting package called “huludesktop-0.9.7-x86_64-1.tgz”.

All flags are not needed, for every situation, and particular ones should be used with care (-S), which can and may frequently wreak havoc on your system.

Take care,

vb

kde-4.3.0 x86_64 for slackware-current

i’ve updated a few of the SlackBuild scripts from the slackware64-current/source/kde.
the packages are uploaded to http://cardinal.lizella.net/~vbatts/kde/kde4-packages/4.3.0/