Querying Installed Package Versions Across An Openstack Cloud

Posted by Craige McWhirter on

AKA: The Joy of juju run

Package upgrades across an OpenStack cloud do not always happen at the same time. In most cases they may happen within an hour or so across your cloud but for a variety reasons, some upgrades may be applied inconsistently, delayed or blocked on some servers.

As these packages may be rolling out a much needed patch or perhaps carrying a bug, you may wish to know which services are impacted in fairly short order.

If your OpenStack cloud is running Ubuntu and managed by Juju and MAAS, here's where juju run can come to the rescue.

For example, perhaps there's an update to the Corosync library libcpg4 and you wish to know which of your HA clusters have what version installed.

From your Juju controller, create a list of servers managed by Juju:

Juju 1.x:

$ juju stat --format tabular > jsft.out

Now you could fashion a query like this, utilising juju run:

$ for i in $(egrep -o '[a-z]+-hacluster/[0-9]+' jsft.out | cut -d/ -f1 | sort -u);
do juju run --timeout 30s --service $i "dpkg-query -W -f='\${Version}' libcpg4" | \
python -c 'import yaml,sys;print("\n".join(["{} == {}".format(y["Stdout"], y["UnitId"]) for y in yaml.safe_load(sys.stdin)]))';
done

The output returned will look something like this:

2.3.3-1ubuntu4 == ceilometer-hacluster/1
2.3.3-1ubuntu4 == ceilometer-hacluster/0
2.3.3-1ubuntu4 == ceilometer-hacluster/2
2.3.3-1ubuntu4 == cinder-hacluster/0
2.3.3-1ubuntu4 == cinder-hacluster/1
2.3.3-1ubuntu4 == cinder-hacluster/2
2.3.3-1ubuntu4 == glance-hacluster/3
2.3.3-1ubuntu4 == glance-hacluster/4
2.3.3-1ubuntu4 == glance-hacluster/5
2.3.3-1ubuntu4 == keystone-hacluster/1
2.3.3-1ubuntu4 == keystone-hacluster/0
2.3.3-1ubuntu4 == keystone-hacluster/2
2.3.3-1ubuntu4 == mysql-hacluster/1
2.3.3-1ubuntu4 == mysql-hacluster/2
2.3.3-1ubuntu4 == mysql-hacluster/0
2.3.3-1ubuntu4 == ncc-hacluster/1
2.3.3-1ubuntu4 == ncc-hacluster/0
2.3.3-1ubuntu4 == ncc-hacluster/2
2.3.3-1ubuntu4 == neutron-hacluster/2
2.3.3-1ubuntu4 == neutron-hacluster/1
2.3.3-1ubuntu4 == neutron-hacluster/0
2.3.3-1ubuntu4 == osd-hacluster/0
2.3.3-1ubuntu4 == osd-hacluster/1
2.3.3-1ubuntu4 == osd-hacluster/2
2.3.3-1ubuntu4 == swift-hacluster/1
2.3.3-1ubuntu4 == swift-hacluster/0
2.3.3-1ubuntu4 == swift-hacluster/2

Juju 2.x:

$ juju status > jsft.out

Now you could fashion a query like this:

$ for i in $(egrep -o 'hacluster-[a-z]+/[0-9]+' jsft.out | cut -d/ -f1 |sort -u);
do juju run --timeout 30s --application $i "dpkg-query -W -f='\${Version}' libcpg4" | \
python -c 'import yaml,sys;print("\n".join(["{} == {}".format(y["Stdout"], y["UnitId"]) for y in yaml.safe_load(sys.stdin)]))';
done

The output returned will look something like this:

2.3.5-3ubuntu2 == hacluster-ceilometer/1
2.3.5-3ubuntu2 == hacluster-ceilometer/0
2.3.5-3ubuntu2 == hacluster-ceilometer/2
2.3.5-3ubuntu2 == hacluster-cinder/1
2.3.5-3ubuntu2 == hacluster-cinder/0
2.3.5-3ubuntu2 == hacluster-cinder/2
2.3.5-3ubuntu2 == hacluster-glance/0
2.3.5-3ubuntu2 == hacluster-glance/1
2.3.5-3ubuntu2 == hacluster-glance/2
2.3.5-3ubuntu2 == hacluster-heat/0
2.3.5-3ubuntu2 == hacluster-heat/1
2.3.5-3ubuntu2 == hacluster-heat/2
2.3.5-3ubuntu2 == hacluster-horizon/0
2.3.5-3ubuntu2 == hacluster-horizon/1
2.3.5-3ubuntu2 == hacluster-horizon/2
2.3.5-3ubuntu2 == hacluster-keystone/0
2.3.5-3ubuntu2 == hacluster-keystone/1
2.3.5-3ubuntu2 == hacluster-keystone/2
2.3.5-3ubuntu2 == hacluster-mysql/0
2.3.5-3ubuntu2 == hacluster-mysql/1
2.3.5-3ubuntu2 == hacluster-mysql/2
2.3.5-3ubuntu2 == hacluster-neutron/0
2.3.5-3ubuntu2 == hacluster-neutron/2
2.3.5-3ubuntu2 == hacluster-neutron/1
2.3.5-3ubuntu2 == hacluster-nova/1
2.3.5-3ubuntu2 == hacluster-nova/2
2.3.5-3ubuntu2 == hacluster-nova/0

You can of course substitute libcpg4 in the above query for any package that you need to check.

By far and away my most favourite feature of Juju at present, juju run reminds me of knife ssh, which is unsurprisingly one of my favourite features of Chef.

Resolving a Partitioned RabbitMQ Cluster with JuJu

Posted by Craige McWhirter on

On occasion, a RabbitMQ cluster may partition itself. In a OpenStack environment this can often first present itself as nova-compute services stopping with errors such as these:

ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._sync_power_states: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c
...
TRACE nova.openstack.common.periodic_task self._raise_timeout_exception(msg_id)
TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 218, in _raise_timeout_exception
TRACE nova.openstack.common.periodic_task 'Timed out waiting for a reply to message ID %s' % msg_id)
TRACE nova.openstack.common.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c

Merely restarting the stopped nova-compute services will not resolve this issue.

You may also find that querying the rabbitmq service may either not return or take an awful long time to return:

$ sudo rabbitmqctl -p openstack list_queues name messages consumers status

...and in an environment managed by juju, you could also see JuJu trying to correct the RabbitMQ but failing:

$ juju stat --format tabular | grep rabbit
rabbitmq-server                       false local:trusty/rabbitmq-server-128
rabbitmq-server/0           idle   1.25.13.1 0/lxc/12 5672/tcp 192.168.7.148
rabbitmq-server/1   error   idle   1.25.13.1 1/lxc/8  5672/tcp 192.168.7.163   hook failed: "config-changed"
rabbitmq-server/2   error   idle   1.25.13.1 2/lxc/10 5672/tcp 192.168.7.174   hook failed: "config-changed"

You should now run rabbitmqctl cluster_status on each of your rabbit instances and review the output. If the cluster is partitioned, you will see something like the below:

ubuntu@my_juju_lxc:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@192-168-7-148' ...
[{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163',
                'rabbit@192-168-7-174']}]},
 {running_nodes,['rabbit@192-168-7-174','rabbit@192-168-7-148']},
 {partitions,[{'rabbit@192-168-7-174',['rabbit@192-168-7-163']},
               {'rabbit@192-168-7-148',['rabbit@192-168-7-163']}]}]
...done.

You can clearly see from the above that there are two partitions for RabbitMQ. We need to now identify which of these is considered the leader:

maas-my_cloud:~$ juju run --service rabbitmq-server "is-leader"
- MachineId: 0/lxc/12
  Stderr: |
  Stdout: |
    True
  UnitId: rabbitmq-server/0
- MachineId: 1/lxc/8
  Stderr: |
  Stdout: |
    False
  UnitId: rabbitmq-server/1
- MachineId: 2/lxc/10
  Stderr: |
  Stdout: |
    False
  UnitId: rabbitmq-server/2

As you see above, in this example machine 0/lxc/12 is the leader, via it's status of "True". Now we need to hit the other two servers and shut down RabbitMQ:

# service rabbitmq-server stop

Once both services have completed shutting down, we can resolve the partitioning by running:

$ juju resolved -r rabbitmq-server/<whichever is leader>

Substituting <whichever is leader> for the machine ID identified earlier.

Once that has completed, you can start the previously stopped services with the below on each host:

# service rabbitmq-server start

and verify the result with:

$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@192-168-7-148' ...
[{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163',
                'rabbit@192-168-7-174']}]},
 {running_nodes,['rabbit@192-168-7-163','rabbit@192-168-7-174',
                 'rabbit@192-168-7-148']},
 {partitions,[]}]
...done.

No partitions \o/

The JuJu errors for RabbitMQ should clear within a few minutes:

$ juju stat --format tabular | grep rabbit
rabbitmq-server                       false local:trusty/rabbitmq-server-128
rabbitmq-server/0             idle   1.25.13.1 0/lxc/12 5672/tcp 19 2.168.1.148
rabbitmq-server/1   unknown   idle   1.25.13.1 1/lxc/8  5672/tcp 19 2.168.1.163
rabbitmq-server/2   unknown   idle   1.25.13.1 2/lxc/10 5672/tcp 192.168.1.174

You should also find the nova-compute instances starting up fine.

First Look at Snaps

Posted by Craige McWhirter on

I've belatedly come to have a close up look at both Ubuntu Core (Snappy), Snaps and the Snappy package manager.

The first pass was to rebuild my rack of Raspberry Pi's from Debian armhf to Ubuntu Core for the Raspberry Pi.

Rack'o'Pi's)

This proved to be the most graceful install I've ever had on any hardware, ever. No hyperbole: boot, authenticate, done. I repeated this for all six Pi's in such a short time frame that I was concerned I'd done something wrong. Your SSH keys are already installed, you can log in immediately and just get on with it.

Which is where snaps come into play.

Back on my laptop, I followed the tutorial Create Your First Snap which uses GNU Hello as an example snap build and finishes with a push to the snap store at snapcraft.io.

I then created a Launchpad Repo, related a snap package, told it to build for armhf and amd64 and before long, I could install this snap on both my laptop and the Pi's.

Overall this was a pretty impressive and graceful process.

Japanese House in Python for Minecraft

Posted by Craige McWhirter on

I have kids that I'm teaching to hack. They started of on Scratch (which is excellent) and are ready to move up to Python. They also happen to be mad Minecraft fans, so now they're making their way through Adventures in Minecraft.

As I used Scratch when they were, I'm also hacking in Python & Minecraft as they are. It helps if I hit the bumps and hurdles before they do, as well as have a sound handle on the problems they're solving.

Now I've branched out from the tutorial and I'm just having fun with it and leaving behind code the kids can use, hack whatever. This code is in my minecraft-tools repo (for want of a better name). It's just a collection of random tools I've written for Minecraft aren't quite up to being their own thing. I expect this will mostly be a collection of python programs to construct things inside Minecraft via CanaryMod and CanaryRaspberryJuicePlugin.

The first bit of code to be shaken out of the tree is japanese_house.py which produces a Minecraft interpretation of a classic Japanese house. Presently it only produces the single configuration that is little more than an empty shell.

Japanese House (day) Japanese House
(night)

I intend to add an interior fit out plus a whole bunch of optional configurations you can set at run time but for now it is what it is, as I'm going to move onto writing geodesic domes and transport | teleport rings (as per the Expanse, which will lead to eventually coding a TARDIS, that will you know, be actually bigger on the inside ;-)

Solarized xmobar, trayer and dmenu for xmonad

Posted by Craige McWhirter on

Like many other people, Ethan Schoonover's precision colours for machines and people, known as Solarized is my preferred colour pallet, especially as I spend most of my time in terminals.

I've "Solarized" most things in my work environment except my window manager, xmonad. Well to be more specific, the xmobar, trayer and dmenu applications that I use within xmonad.

xmobar screenshot

The commits:

Modifications were made to xmonad.hs, .xmobarrc and trayer in .xsession that switches out the existing and default colours for those I've selected from the Solarized palette. I now have a fully Solarized window manager that fits in much better with the rest of my workspace.

Relevant code snippets:

xmonad.hs

                , logHook = dynamicLogWithPP $ xmobarPP
                    { ppOutput = hPutStrLn xmproc
                    , ppCurrent = xmobarColor "#859900" "" . wrap "[" "]"
                    , ppVisible = xmobarColor "#2aa198" "" . wrap "(" ")"
                    , ppLayout = xmobarColor "#2aa198" ""
                    , ppTitle = xmobarColor "#859900" "" . shorten 50
                    }
            , modMask = mod4Mask -- Rebind Mod to the Windows key
            --, borderWidth = 1
            } `additionalKeys`
                    -- Custom dmenu launcher
                    [ ((mod4Mask, xK_p ), spawn "exe=`dmenu_path | dmenu -fn \"Open Sans-10\" -p \"λ:\" -nb \"#073642\" -nf \"#93a1a1\" -sb \"#002b36\" -sf \"#859900\"` && eval \"exec $exe\"")

.xmobarrc

-- Appearance
  font = "xft:OpenSans:size=10:antialias=true"
, bgColor = "#073642"
, fgColor = "#93a1a1"

...

--Plugins
, commands =
    -- CPU Activity Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- cpu core temperature monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Memory Usage Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Battery Monitor
    ...
                         , "--low"      , "#dc322f"
                         , "--normal"   , "#859900"
                         , "--high"     , "#2aa198"

                         , "--" -- battery specific options
                                   -- discharging status
                                   , "-o"   , "<left>% (<timeleft>)"
                                   -- AC "on" status
                                   , "-O" , "<fc=#2aa198>Charging</fc>"
                                   -- charged status
                                   , "-i"   , "<fc=#859900>Charged</fc>"
    ...
    -- Weather Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Network Activity Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Time and Date Display
    , Run Date           "<fc=#268bd2>%a %b %_d %H:%M</fc>" "date" 10
    ...

.xsession

exec /usr/bin/trayer --edge top --align right --SetDockType true --SetPartialStrut true --expand true --width 10 --transparent true --alpha 0 --tint 0x073642 --height 22 &

That at least implements the colours I selected from Solarized and should be a good starting point for anyone else.

Adding a Docker Runner to GitLab

Posted by Craige McWhirter on

In my particular scenario, I need to run both docker and docker-compose to test and build our changes. The first step to achieving this is to add an appropriate GitLab runner.

We especially need to run a privileged runner to make this happen.

Assuming that GitLab Runner has already been successfully installed, head to Admin -> Runner in the webUI of your GitLab instance and note your Registration token.

From a suitable account on your GitLab instance register a shared runner:

% sudo /usr/bin/gitlab-ci-multi-runner register --docker-privileged \
    --url https://gitlab.my.domain/ci \
    --registration-token REGISTRATION_TOKEN \
    --executor docker \
    --description "My Docker Runner" \
    --docker-image "docker:latest" \

Your shared runner should now be ready to run.

This applies to self-hosting a GitLab instance. If you are using the gitlab.com hosted services, a suitable runner is already supplied.

There are many types of executors for runners, suiting a variety of scenarios. This example's scenario is that both GitLab and the desired runner are on the same instance.

Have It Your Way: Maximizing Drive-Thru Contributions - PyConAu 2016

Posted by Craige McWhirter on

by VM (Vicky) Brasseur.

Slides.

Vicky talked about the importance non-committing contributors but the primary focus is on committing contributors due to time limits.

Covered the different types of drive-thru contributors and why they show up.

  • Scratching an itch.
  • Unwilling / Unable to find an alternative to this project
  • They like you.

Why do they leave?

  • Itch has been sratched.
  • Not enough time.
  • No longer using the project.
  • Often a high barrier to contribution.
  • Absence of appreciation.
  • Unpleasant people.
  • Inappropriate attribution.

Disadvantages

  • It takes more time to help them land patches
    • Reluctance to help them "as they're not community".

It appears to be that many project see community as the foundation but Vicky contended it is contributors.

More drive-thru contributors are a sign of a healthy project and can lead to a larger community.

Advantages:

  • Have better processes in place.
  • Faster patch and release times.
  • More eyes and shallower bugs
  • Better community, code and project reputation.

Leads to a healthier overall project.

Methods for Maxmising drive-thru contributions:

Documentation!

  • give your project super powers.
  • Scales!
  • Ensures efficient and successful contributions.
  • Minimises questions.
  • Standardises processes.
  • Vicky provided a documentation quick start guide.

Mentoring!

  • Code review.
  • "Office hours" for communication.
  • Hackfests.
  • New contributor events.

Process improvements!

  • Tag starter bugs
  • Contributor SLA
  • Use containers / VM of dev environment

Culture!

  • Value contributions and contributors
  • Culture of documentation
  • Default to assistance

Outreach! * Gratitude * Recognition * Follow-up!

Institute the "No Asshole" rule.

PyConAu 2016

Keynote - Python All the Things - PyConAu 2016

Posted by Craige McWhirter on

by Russell Keith-Magee.

Russell spoke about porting Python to mobile devices. CPython being written in C enables it to leverage the supported platforms of the C language and be compiled a wide range of platforms.

There was a deep dive in the options and pitfalls when selecting a method to and implementing Python on Android phones.

Ouroboros is a pure Python implementation of the Python standard library.

Most of the tools discussed are at an early stage of development.

Why?

  • Being able to run on new or mobile platforms addresses an existential threat.
  • The threat also presents an opportunity to grown, broaden and improve Python.
  • Wants Python to be a "first contact" language, like (Visual) Basic once was.
  • Unlike Basic, Python also support very complex concepts and operations.
  • Presents an opportunity to encourage broader usage by otherwise passive users.
  • Technical superiority is rarely enough to guarantee success.
  • A breadth of technical domains is required for Python to become this choice.
  • Technical problems are the easiest to solve.
  • Te most difficult problems are social and community and require more attention.

Russell's will be putting his focus into BeeWare and related projects.

Fortune favours the prepared mind

(Louis Pasteur)

PyConAu 2016

Python for science, side projects and stuff! - PyConAu 2016

Posted by Craige McWhirter on

By Andrew Lonsdale.

  • Talked about using python-ppt for collaborating on PowerPoint presentations.
  • Covered his journey so far and the lessons he learned.
  • Gave some great examples of re-creating XKCD comics in Python (matplotlib_venn).
  • Claimed the diversion into Python and Matplotlib has helped is actual research.
  • Spoke about how using Python is great for Scientific research.
  • Summarised that side projects are good for Science and Python.
  • Recommended Elegant SciPy
  • Demo's using Emoji to represent bioinformatics using FASTQE (FASTQ as Emoji).

PyConAu 2016

MicroPython: a journey from Kickstarter to Space by Damien George - PyConAu 2016

Posted by Craige McWhirter on

Damien George.

Motivations for MicroPython:

  • To provide a high level language to control sophisticated micro-controllers.
  • Approached it as an intellectually stimulating research problem.
  • Wasn't even sure it was possible.
  • Chose Python because:
    • It was a high level language with powerful features.
    • Large existing community.
    • Naively thought it would be easy.
    • Found Python easy to learn.
    • Shallow but long learning curve of python makes it good for beginners and advanced programmers.
    • Bitwise operaitons make it usefult for micro-controllers.

Why Not Use CPython?

  • CPython pre-allocates memory, resulting in inefficient memory usage which is problematic for low RAM devices like micro controllers.

Usage:

  • If you know Python, you know MicroPython - it's implemented the same

Kickstarter:

Damien covered his experiences with Kickstarter.

Internals of MicroPython:

  • Damien covered the parser, lexer, compiler and runtime.
  • Walked us through the workflows of the internals.
  • Spoke about object represntation and the three machine word object forms:
    • Integers.
    • Strings.
    • Objects.
  • Covered the emitters:
    • Bytecode.
    • Native (machine code).
    • Inline assembler.

Coding Style:

Coding was more based on a physicist trying to make things work, than a computer engineer.

  • There's a code dashboard
  • Hosted on GitHub
  • Noted that he could not have done this without the support of the community.

Hardware:

Listed some of the micro controller boards that it runs on ad larger computers that currently run OpenWRT.

Spoke about the BBC micron:bit project. Demo'd speech synthesis and image display running on it.

MicroPython in Space:

Spoke about the port to LEON / SPARC / RTEMS for the European Space agency for satellite control, particularly the application layer.

Damien closed with an overview of current applications and ongoing software and hardware development.

Links:

micropython.org forum.micropython.org github.com/micropython

PyConAu 2016