Welcome to purpleidea.com on purpleidea.com

My perspectives on AI/LLMs

Wed, 24 Jun 2026 04:35:00 -0400

There’s a lot of talk about “AI” and “LLMs” recently. Here are my thoughts, experiences, and perspectives about all that. There seems to be a growing rift in the software industry about these tools. I’m publishing this post to make my opinions as clear as possible. Of note, I’ve provided specific git commit examples of some interesting results I’ve had when using these models.

Watercolour painted by a human artist.

Background:

For a very long time, I have been, and still am, a strong Free Software advocate. Everyone who knows me knows what I stand for. I’m about actions and results, not just talk. I’ve spent a great amount of personal time and money writing Free Software because I believe it’s an important thing that our society needs. Without the four freedoms, we empower hegemons to control our computing, our data and our lives. AI and LLMs will accelerate our demise or uplift us. Let’s be smart about this.

The switch as an ordinary GNU/Linux server

Thu, 09 Apr 2026 02:23:52 -0400

In 2014 I wrote an article about the switch as an ordinary GNU/Linux server. This had been my dream since my early days using Linux to build routers and I didn’t know why we didn’t use the same management interfaces on switches as we did on Linux. The folks at Cumulus Networks made this a reality and in 2014 they were kind enough to give me access to some hardware to test things out.

10 years of mgmt

Thu, 25 Sep 2025 01:07:56 -0400

I publicly released mgmt to the internet 10 years ago. I can’t believe it’s been 10 years. In this post I’ll talk about:

The new mgmt company
A one-time new customer offer
Unexpected wins
Past failures
Some history
A new release

And more…

The company:

I’ve officially started a company. I’ve been working on this for some time now, and it’s time to let you all know about it officially. As you already know (mgmt, purpleidea, etc…) I’ve never been great at naming. The company is called:

Better git status

Mon, 04 Aug 2025 21:10:30 -0400

I’ve been using git status for quite a long time now. Let’s make it better.

My old aliases:

My ~/.gitconfig has had:

[alias]
	s = status
	d = diff
	l = log --show-signature

for as long as I can remember. They’ve served me well.

The problem:

In one of my projects, I occasionally have a bunch of untracked files, particularly when I’m hacking on something new that isn’t committed yet. This may include notes, new tests, and so on. My terminal is usually between 24 and 37 lines tall. (Eg: run tput lines.) If there’s a lot of untracked stuff here, I’ll quickly go over this limit and have to scroll to see what’s staged and so on.

Running etcd over SSH

Wed, 14 May 2025 19:45:00 -0400

Want to tunnel etcd traffic over SSH? Here’s a pitfall I ran into.

The problem:

Mgmt supports tunnelling the etcd traffic over SSH. This is quite useful, because it means that for it to connect to etcd, it doesn’t need to open up any new ports other than 22.

You can run the agent like so:

mgmt run --ssh-url=etcd@etcdserver:22 --seeds=http://127.0.0.1:2379 --no-server --no-magic empty

and it just works!

Working around an iPXE issue

Thu, 23 Jan 2025 21:34:56 -0500

iPXE is a terrific tool, but despite their very respectable documentation, it has some rough edges, and it still feels like an “in crowd” kind of thing, where you have to learn the common tricks by knowing someone.

The problem:

I’ve been building a provisioning tool in mgmt and instead of always relying on distro-specific boot images, I decided to use iPXE to handle the early boot, and have it hand-off to the kernel installer.

Modules and imports in mgmt

Tue, 03 Dec 2024 02:56:37 -0500

Mgmt modules are used for structuring your code across multiple files. This keeps things readable and allows for code reusability. For example, it should be easy to write a single module which can be effectively used by many different parties. To me, it never felt like code reuse was a core design objective with other tools. In mgmt it has been since the very beginning. I’m very proud of our module system, so let me introduce you to it properly.

Mgmt Configuration Language: Functions

Fri, 22 Nov 2024 18:16:00 -0500

It’s been a little while since I introduced the Mgmt Configuration Language. I originally wrote this article in 2019, and as I was working on it alongside the code for functions, when I realized that lambdas didn’t work properly. It took some time to finally solve that properly. Since then I’ve been working to get mgmt production ready. It’s properly useful for production now, and so it’s time for me to catch up on my documentation. I’ve revived and updated this article, let’s go…

Better dmesg in five minutes

Sun, 17 Nov 2024 15:52:00 -0500

I last wrote about dmesg in 2016. It has mostly not changed since then, but I’ve changed my setup slightly. Here’s a short article about what I did so that you can do it too, and so that I can remember for the next time.

Previously:

Previous I had a bash alias which looked like this:

alias dmesg='dmesg --follow || dmesg'

since some machines didn’t yet support the --follow flag.

A new provisioning tool built with mgmt

Wed, 27 Mar 2024 16:58:57 -0400

Today I’m announcing a new type of provisioning tool. This is both the culmination of a long road, and the start of a new era. Please read on for all of the details. Feel free to skip to the relevant sections you’re interested in if you don’t want all of the background.

Ten years:

The vision for this specific tool started around ten years ago. Previously, as a sysadmin, I spent a lot of my time using a configuration management tool called puppet. I eventually realized and accepted the shortcomings of the tool after a failed attempt to combine puppet and some python scripts into a provisioning and clustering tool that I was trying to build for greenfield environments. The idea is that it would bootstrap a whole suite of infrastructure services for a new data centre.

Bash $PATH filtering

Sat, 02 Mar 2024 19:54:00 -0500

As most modern GNU+Linux distro users already know, you get a lot of tools included for free! Many of these may clutter up your $PATH and make bash tab completion more difficult than it has to be. Here’s a way to improve this!

A mess:

Here’s what I see when I tab-complete cd<TAB>:

james@computer:~$ cd
cd                 cddb_query         cd-info            cd-read
cd-convert         cd-drive           cd-it8             cdrecord
cd-create-profile  cd-fix-profile     cdmkdir            cdtmpmkdir
cdda-player        cd-iccdump         cd-paranoia

I genuinely only use three of those commands.

Upcoming Speaking and Events

Wed, 31 Jan 2024 09:57:30 -0500

There are three upcoming mgmtconfig events happening within the next seven days!

mgmt @ FOSDEM:

On February 3rd, I’ll be giving a talk in the golang devroom at FOSDEM. Please come early (1hr+) if you want to get a seat, usually this room is especially full. This happens in Brussels, but it should be recorded if there aren’t any technical glitches.

(I got rejected for the main track, but it’s my fault because I need to make my work more generally appealing first.)

Git branch filtering

Sun, 28 Jan 2024 17:04:30 -0500

If you have more than one git branch up in the air at the same time, having a system to track them and filter them is essential. Here are the stages I went through with git branch.

xkcd reminds us that we're insane

Prefixes:

My first tip involves using a memorable prefix. For new features I name my branches with feat/. For example feat/my-new-feature. Yes the slash and the dashes are treated like any regular character. I also commonly use bug/. You can see all of the branches with git branch -a.

Five minute hacks: Swapping left and right headphone audio in wireplumber

Wed, 22 Nov 2023 17:25:00 -0500

I’m on Fedora 37 at the moment. My headphone audio is backwards. How do we fix it? I’ve apparently never found the right magic until now.

The Setup:

I have a crappy “couch computer”. It’s used for casual internet browsing and video watching. It has an analog audio out which goes to my speakers. I’d like to use headphones for when I don’t want to annoy my neighbours. I have a pair of wireless headphones that I got for free, but the battery doesn’t last longer than an hour. I also have an old pair of analog headphones that work okay and don’t run out of battery! I got a free USB to analog dongle, but for some reason the left and right audio channels are reversed!

One minute hacks: Saving time inserting images in Libreoffice

Wed, 02 Aug 2023 13:35:00 -0400

You’ve probably used libreoffice. You opened a document, and proceeded to insert an image into the text. It wiggles all over and never rests where you want it to.

The solution:

You most likely want the “anchor as character” option. Right-click on the newly inserted image, go to the Anchor menu item, and choose As Character. You will have to do this for each image you insert which is annoying…

UEFI booting, and with a new kernel module

Mon, 24 Jul 2023 17:22:22 -0400

Picture this. You’ve got a supermicro server. It’s got eight hot-swappable 3.5" HDD’s. You want to use all of those for storage, so you buy a nifty add-on that let’s you add four additional 2.5" SSD’s. Those are great for your OS. That’s a great way to keep your data separate. The motherboard only supports eight drives, so you’ll need an HBA for the additional four… What are some catches and things you need to watch out for?

Improving git diff output in 5 min

Thu, 01 Jun 2023 18:20:00 -0400

Everyone uses git diff. But there’s that odd annoyance that you’ve probably seen before but are so used to that you’ve forgotten about it. The a/ and b/ prefixes…

Example:

james@computer:~/code/mgmt (feat/cool-feature)$ git diff
diff --git a/lang/funcs/core/os/readfile_func.go b/lang/funcs/core/os/readfile_func.go
index 206ba798d..48ac29dc7 100644
--- a/lang/funcs/core/os/readfile_func.go
+++ b/lang/funcs/core/os/readfile_func.go
@@ -228,6 +228,8 @@ func (obj *ReadFileFunc) Stream() error {

 // Close runs some shutdown code for this function and turns off the stream.
 func (obj *ReadFileFunc) Close() error {
+       // TODO: Port to the new ctx-in-Stream close API.
+       // Hello blog readers!
        close(obj.closeChan)
        obj.wg.Wait()     // block so we don't exit by the closure of obj.events
        close(obj.events)

What are they for?

The git diff command shows these to help you differentiate between the source and the destination files. In most cases you are in the midst of looking at and writing code, so you’ll probably instinctively know whether you or something just added or deleted those lines. (Because if you flipped a and b you’d have the opposite diff.)

Blocked select logging (deadlock finding) in golang

Fri, 26 May 2023 01:21:22 -0400

I’ve been writing some massively concurrent code for mgmt lately. The concurrency makes it incredibly bug prone. In this case it’s usually races or deadlocks. I’ve got a new trick to make the process more pleasant.

The Problem:

I’m sure many of you have written some code like this before:

// just a small excerpt
for {
	select {
	case <-someEvent:
		// some awesome thing happened!
	case <-ctx.Done():
		return ctx.Err()
	}
}

And during debugging, you might have gotten frustrated and added some printf’s:

Deadline context test cancellation in golang

Fri, 24 Feb 2023 22:36:00 -0500

I decided to write a fancy test harness in golang today. The test wraps a big internal engine for mgmt and at the top-level it takes a context for cancellation. If you don’t know about the context package, then you should go understand that and then come back here… Don’t feel bad, I had no idea what it was about at first either!

The Problem:

I assumed there would be some way to follow a notification from the test runner down into my test to tell it when it was time to cleanup and exit early… I expected that making my own ^C signal handler wouldn’t be correct, and I (incorrectly) assumed that the interface I’d be looking for would offer a golang context that I could pass into my code. I was wrong, but I was on the right track and it did help me find what I was looking for. Let me explain the subtlety…

Magic typing with shortcuts on Linux

Wed, 16 Jun 2021 00:40:00 -0400

Ever find yourself typing more characters than is necessary when using some annoying application or website? Here’s a quick hack around that.

Intro:

There’s a magic program called xdotool written by Jordan Sissel that solves this problem. I’ve known about his excellent work for years, but only today did I learn he also wrote xdotool. It’s named as such because it tells the X server to do something for you.

USB Audio on a Boombox

Mon, 11 Jan 2021 04:00:00 -0500

In an earlier post, I got USB audio working on my used, manual, 2013 Honda Civic EX. Today I’ll do the same for a Sony ZS-RS60BT boombox.

Here it is in all its used glory after a little cleaning.

Background:

I decided I should get a small portable boombox to practice my dope dance moves. I currently still suck because I think I spend more time computering than out on the floor. In any case, in a stroke of luck, I happened upon a used boombox that was getting discarded due to a broken CD player. Nobody seemed either able or interested in getting USB audio working, so I took it home and gave it a go!

Working at Amazon

Mon, 28 Dec 2020 08:27:35 -0500

TL;DR: I’ve got a new job working for Amazon, in the “Open Source Program Office”. I’m now an Amazoner! More information below…

The Amazon. Source: https://en.wikipedia.org/wiki/Amazon_rainforest

Background:

A little over three months ago, I decided to get back into full-time work. For the past 2.5 years since I left my job at Red Hat, I’ve been mostly working full-time on Mgmt Config and my DevOps/SWE Mentoring Program. To do such things, I had to live off of my savings, which isn’t sustainable long-term. I am thankful that I was able to do so for so long. It was time to dive back in, and here I am.

James is available for hire!

Thu, 10 Sep 2020 08:00:00 -0400

TL;DR: I’m available for hire. Experienced at Linux/Golang/Mentoring/etc. More information below!

A drawing of me coding away.

A photo of me giving a talk to a sold-out crowd.

Background:

Two years ago, I left my job at Red Hat to work on mgmt config full-time. I’m still passionate about this project, and I’m proud of the progress that has been made in the last two years, but it’s time for me to explore new opportunities as well. My years of experience building mgmt has taught me a lot about open source, golang and mentoring others, and I hope this makes me a valuable addition to your team.

Inexpensive Dynamic DNS over IRC

Tue, 01 Sep 2020 17:08:00 -0400

The one thing that I’ve been using for longer than ssh is dynamic dns. In this article I’ll explain how I built an inexpensive, reliable, private alternative to the mainstream tools commonly available today using only golang and IRC.

Background:

I’ve had an account with dyndns.org for a long time. They used to be free, but then they cut off the free tier, but left existing accounts alone, until they eventually cut all the non-payers off entirely. I think we need an inexpensive way of giving up-and-coming GNU+Linux hackers a way to ssh home, and run their irssi sessions, without having to pay. I also wanted something reliable that wouldn’t disappear the way dyndns did.

USB Car Audio in a Civic

Mon, 24 Aug 2020 01:30:00 -0400

Transportation is very important to me. My whole life I’ve never owned a car. I have gotten by with biking, public transit, and various car sharing services. Sadly, biking is incredibly dangerous and stressful, because most cars don’t respect bicycles, and in the winter it’s not very practical, the STM (public transit) offers an incredibly unreliable and unpleasant service, and Car2Go (car sharing) now requires that you use their proprietary smart phone app (you used to be able to just swipe the membership card) so I finally gave up and decided to buy a used car.

Mgmt Configuration Language: Class and Include

Fri, 26 Jul 2019 06:35:00 -0400

It’s been a little over a year since I introduced the Mgmt Configuration Language. A lot has happened since then, and I’d like to introduce some of the missing features that weren’t available when the language was first introduced. If you haven’t already read that post, please start there and come back when you’re finished. In this article we’ll learn about classes.

Classes:

You might remember that the mgmt language called mcl has both statements, and expressions. Statements ultimately produce the resources and edges that make up our output graph, and expressions produce the values (strings, integers, lists, etc) that make up the inputs to those statements.

Upcoming Speaking and Events

Sat, 02 Feb 2019 03:00:00 -0500

mgmt @ FOSDEM:

Later today I’m giving four talks about mgmt at FOSDEM. I’m also giving two more on the following day. (One is a 5 min lightning talk.) I don’t know how I got accepted for so many talks, I just replied to the CFP for every room where I thought I had something interesting to share, and well, here we are!

I got rejected for the main track again, but I guess it’s up to me to make my work more generally appealing first.

A guide to fighting with a Cisco access point

Fri, 21 Dec 2018 07:45:00 -0500

In my hands is a Cisco AIR-CAP-3502-E-A-K9 that I saved from the landfill. After about eight hours of suffering, and what was surely a cancer causing dose of RF-radiation, I now know:

How to convince this access point to provide me with wifi.
Why Cisco products are probably best left in the garbage.

What follows is my setup guide and review for using a Cisco Aironet device. This guide will likely be useful with other models of Cisco devices, but use your own best judgment.

The End of a Red Giant

Tue, 30 Oct 2018 08:39:00 -0400

As I’m sure you’ve now heard, Red Hat is being acquired by IBM.

Reactions have ranged from:

The first two emotions everyone experienced.

to:

The classic humourous response.

to:

The classy response.

to:

The thing we're all curious about.

to everything in between. If someone were to leak me a copy of the memo-list 2018-October.txt.gz mailman archive, I’d sure love to see that! (Of course I’m joking and I wouldn’t actually ask anyone to do this.)

Hyperlinks in GNOME terminal

Fri, 29 Jun 2018 21:45:00 -0400

Over the years I’ve learned about many of the advantages of using a modern terminal and shell. I’m talking about using bash with GNOME terminal on a modern GNU/Linux distribution.

I particularly like switching between the terminal and GUI applications. It’s now even better.

Automatic Hyperlinks:

Occasionally, some program running in your shell might output what looked like a link:

gnome terminal and vte will automatically add links to text that looks like a web hyperlink or an email address

They can be seen when you hover over them, and a right-click will display a useful context menu:

Breaking out of the Slack walled garden

Fri, 22 Jun 2018 13:50:00 -0400

I’m old school cool. Real hackers chat on open, distributed platforms. Most technical discussion can be found on the Libera.chat IRC network. It’s not perfect, but the advantages clearly outweigh the drawbacks.

Recently, I needed to join an existing large “community” on the centralized, proprietary walled garden that is the Slack network.

The Problem:

Connecting to the the Slack server requires that you use either the proprietary client or their proprietary web app. There’s zero chance I will install the former on my machine, and the latter uses an absurd amount of browser memory and CPU. I’m already happy with my irssi experience, and I didn’t want to need another app.

A new sync primitive in golang

Sun, 20 May 2018 18:00:00 -0400

I’ve been working on lots of new stuff in mgmt and I had a synchronization problem that needed solving… Long story short, I built it into a piece of re-usable functionality, exactly like you might find in the sync package. For details and examples, please continue reading…

The Problem:

I want to multicast a signal to an arbitrary number of goroutines. As you might already know, this can already be done with a chan struct{}. You simply close the channel to send the signal, and anyone running a select on that channel will return when it closes.

Leaving Red Hat

Wed, 04 Apr 2018 06:27:35 -0500

I’ve spent about four years at Red Hat, and now it’s time to move on…

TL;DR: had to leave Red Hat and start Patreon to fund mgmt.

What follows is a bit of historical rambling, and some forward looking statements.

Retrospective:

Long-time readers of my blog will know that I was very active in the puppet ecosystem for many years. I learned a lot while writing puppet code, and while building some of my outrageous puppet hacks. Over this time, I had quietly put together some design notes and a prototype for some “future tool” ideas. I had code-named this project “puppet-ng”.

Running `make` from anywhere

Sat, 10 Mar 2018 02:00:00 -0500

Sometimes while I’m deep inside mgmt’s project directory, I want to run an operation from the Makefile which lives in the root! Unfortunately, if you do so while nested, you’ll just get:

james@computer:~/code/mgmt/resources$ make build
make: *** No rule to make target 'build'.  Stop.

The Ten Minute Solution:

I figured I’d hack out a quick solution. What I came up with looks like this:

#!/bin/bash
# James Shubin, 2018
# run `make` in the first directory (or its parent recursively) that it works in

MF='Makefile'	# looks for this file, could look for others, but that's silly
CWD=$(pwd)	# starting here

while true; do
	if [ -e "$MF" ]; then
		make $@		# run make!
		e=$?
		cd "$CWD"	# go home first
		exit $e		# pass on the exit status
	fi

	# stop if we get home
	if [ "$(pwd)" = "$HOME" ]; then
		cd "$CWD"
		exit 2		# the exit status of `make` when it errors
	fi

	#echo "searching..."
	cd ..			# go up one dir
done

You can probably figure it out quite easily from the code, but what this does is it searches upwards until it finds a Makefile, and then runs your command from there!

Mgmt Configuration Language

Mon, 05 Feb 2018 00:00:00 -0500

Mgmt has given us a lot of great features, demos, and insight into how infrastructure automation should be managed. Unfortunately it wasn’t a complete tool yet, because it was missing a crucial piece for its completeness. That all changes today with the addition of: the language.

An ideal language allows us to express easily what is useful for the programming task, and at the same time makes it difficult to write what leads to incomprehensible or incorrect programs.

Hello World!

Mon, 13 Nov 2017 09:18:15 -0500

Hello! This is a new version of my website and blog. If you experience any issues, please let me know!

Happy Hacking, James

You can hire James and his team at m9rx corporation.
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.

Copyleft is Dead. Long live Copyleft!

Tue, 17 Oct 2017 21:22:05 +0000

As you may have noticed, we recently re-licensed mgmt from the AGPL (Affero General Public License) to the regular GPL. This is a post explaining the decision and which hopefully includes some insights at the intersection of technology and legal issues.

Disclaimer:

I am not a lawyer, and these are not necessarily the opinions of my employer. I think I’m knowledgeable in this area, but I’m happy to be corrected in the comments. I’m friends with a number of lawyers, and they like to include disclaimer sections, so I’ll include this so that I blend in better.

Extracting movies from libreoffice

Wed, 12 Jul 2017 10:09:46 +0000

I have a short movie that I imported into a libreoffice presentation. I wanted a copy of that movie back, but I couldn’t figure out how to extract a copy. In desperation, I figured I’d try opening the file with file-roller, the GNOME archive manager.

james@computer:/tmp$ file mgmt-berlin-osdc-17may2017.odp 
mgmt-berlin-osdc-17may2017.odp: OpenDocument Presentation
james@computer:/tmp$ mkdir out
james@computer:/tmp$ file-roller -f mgmt-berlin-osdc-17may2017.odp -e out/
[snip]
james@computer:/tmp$ cd out/
james@computer:/tmp/out$ ls
Configurations2/ Media/ meta.xml Pictures/ styles.xml
content.xml META-INF/ mimetype settings.xml Thumbnails/
james@computer:/tmp/out$

To my amazement, this worked perfectly! I found my video in the Media/ folder, and out it came! You can do this entirely with the GUI if you prefer:

Declarative vs. Imperative paradigms

Fri, 05 May 2017 17:35:17 +0000

Recently, while operating two different remote-controlled appliances, I realized that it was high time for a discussion about declarative and imperative paradigms. Let’s start by looking at the two remotes:

Two different "remotes". The one on the left operates a television, and the one on the right controls a central heating and cooling system.

At first glance you will notice that one of these remotes is dark, and the other is light. You might also notice that my photography skills are terrible. Neither of these facts is very important to the discussion at hand. Is there anything interesting that you can infer?

Osmocom femtocell un-boxing

Sun, 30 Apr 2017 17:56:13 +0000

LaForge and the fine folks at Osmocom (Sysmocom) recently had a femtocell giveaway. I didn’t expect to have much time to hack on things, but they were still quite generous in sending me one. It arrived, and I took some un-boxing photos for anyone who is curious.

A box arrived in the mail...

Which recurses into an inner box...

Inner box is box like.

Finally... The unit is displayed.

Here it is in all its glory.

It came complete with all the cables and power adapters necessary.

The sysmocom folks even included some special test sim cards so that you can use the femtocell with your own phones.

I’m very excited about the work the project is doing in their efforts to create and support free software telephony. If you are a telecom, please check out the various Sysmocom products and services available.

Metaparameters in mgmt

Wed, 01 Mar 2017 16:11:59 +0000

In mgmt we have meta parameters. They are similar in concept to what you might be familiar with from other tools, except that they are more clearly defined (in a single struct) and vastly more powerful.

In mgmt, a meta parameter is a parameter which is codified entirely in the engine, and which can be used by any resource. In contrast with Puppet, require/before are considered meta parameters, whereas in mgmt, the equivalent is a graph edge, which is not a meta parameter. [1]

Faster golang builds

Sun, 26 Feb 2017 17:02:35 +0000

I’ve been hacking in golang since before version 1.4, and the speed at which my builds finished has been mostly trending downwards. Let’s look into the reasons and some fixes. TL;DR click-bait title: “Get 4x faster golang builds with this one trick!”.

Here are the three reasons my builds got slower:

The compiler

Before version 1.5, the compiler was written in C but with that release, it moved to being pure golang. This unfortunately reduced build performance quite measurably, even though it was the right decision for the project.

Ten minute hacks: Process pause & resume

Fri, 06 Jan 2017 05:57:08 +0000

I’m old school and still rocking an old X220 laptop because I didn’t like the new ones. My battery life isn’t as great as I’d like it to be, but it gets worse when some “webapp” (which I’d much rather have as a native GTK+ app) causes Firefox to rev my CPU with their websocket (hi gmail!) poller.

This seems to happen most often on planes or when I’m disconnected from the internet. Since it’s difficult to know which tab is the offending one, and since I might want to keep that tabs state anyway, I decided to write a little shell script to pause and resume misbehaving processes.

Send/Recv in mgmt

Wed, 07 Dec 2016 07:00:20 +0000

I previously published “A revisionist history of configuration management”. I meant for that to be the intro to this article, but it ended up being long enough that it deserved a separate post. I will explain Send/Recv in this article, but first a few clarifications to the aforementioned article.

Clarifications

I mentioned that my “revisionist history” was inaccurate, but I failed to mention that it was also not exhaustive! Many things were left out either because they were proprietary, niche, not well-known, of obscure design or simply for brevity. My apologies if you were involved with Bcfg2, Bosh, Heat, Military specifications, SaltStack, SmartFrog, or something else entirely. I’d love it if someone else wrote an “exhaustive history”, but I don’t think that’s possible.

A revisionist history of configuration management

Wed, 30 Nov 2016 23:59:49 +0000

I’ve got a brand new core feature in mgmt called send/recv which I plan to show you shortly, but first I’d like to start with some background.

History

This is my historical perspective and interpretation about the last twenty years in configuration management. It’s likely inaccurate and slightly revisionist, but it should be correct enough to tell the design story that I want to share.

Sometime after people started to realize that writing bash scripts wasn’t a safe, scalable, or reusable way to automate systems, CFEngine burst onto the scene with the first real solution to this problem. I think it was mostly all quite sane, but it wasn’t a tool which let us build autonomous systems, so people eventually looked elsewhere.

Remote execution in mgmt

Fri, 07 Oct 2016 15:33:17 +0000

Bootstrapping a cluster from your laptop, or managing machines without needing to first setup a separate config management infrastructure are both very reasonable and fundamental asks. I was particularly inspired by Ansible’s agent-less remote execution model, but never wanted to build a centralized orchestrator. I soon realized that I could have my ice cream and eat it too.

Prior knowledge

If you haven’t read the earlier articles about mgmt, then I recommend you start with those, and then come back here. The first and fourth are essential if you’re going to make sense of this article.

mgmt has a logo

Fri, 30 Sep 2016 08:41:06 +0000

The mgmt config project got a logo! The full commit is here. Thanks to Sarah Jane Cox for creating it.

Happy Hacking,

James

PS: I might have a few stickers to give out too! Ask me next time you see me if you’d like one! Alternatively, use the artwork to make your own and share with your friends!

Live dmesg following

Mon, 29 Aug 2016 09:00:39 +0000

All good sysadmins eventually learn about using tail -F to tail files. Yes upper-case F is superior.

Around the time I wrote that article, I remember wanting to stream dmesg output too! The functionality wasn’t available without some sort of polling hack, but it turns out that kernel support for this actually landed around the same time in version 3.5.0!

Most GNU/Linux distros are probably running a new enough version by now, and you can now dmesg –follow (or dmesg -w):

Seen in downtown Montreal...

Wed, 03 Aug 2016 01:59:23 +0000

The Technical Blog of James was seen on an outdoor electronic display in downtown Montreal! Thanks to one of my readers for sending this in.

I guess the smart phone revolution is over, and people are taking to reading my articles on bigger screens! The "poutine" is decent proof that this is probably Montreal.

If you’ve got access to a large electronic display, put up the blog, snap a photo, and send it my way! I’ll post it here and send you some random stickers!

Ten minute hacks: Hacking airplane headphones

Mon, 18 Jul 2016 16:41:47 +0000

I was stuck on a 14 hour flight last week, and to my disappointment, only one of the two headphone speakers were working. The plane’s media centre has an audio connector that looks like this:

Someone should consider probing this USB port.

The hole to the left is smaller than a 3.5mm headphone jack, and designed for a proprietary headphone connector that I didn’t have, and the two holes to the right are part of a different proprietary connector which match with the cheap airline headphones to provide the left and right audio channels.

Automatic clustering in mgmt

Mon, 20 Jun 2016 12:28:51 +0000

In mgmt, deploying and managing your clustered config management infrastructure needs to be as automatic as the infrastructure you’re using mgmt to manage. With mgmt, instead of a centralized data store, we function as a distributed system, built on top of etcd and the raft protocol.

In this article, I’ll cover how this feature works.

Foreword:

Mgmt is a next generation configuration management project. If you haven’t heard of it yet, or you don’t remember why we use a distributed database, start by reading the previous articles:

Upcoming speaking In Hong Kong and South Africa

Mon, 20 Jun 2016 04:08:59 +0000

I’m thrilled to tell you that I’ll be speaking about mgmt in Hong Kong and South Africa. It will be my first time to both countries and my first time to Asia and Africa!

In Hong Kong I’ll be speaking at HKOSCon2016.

In South Africa I’ll be speaking at DebConf16.

I’m looking forward to meeting with many of the hard-working Debian hackers, and collaborating with them to build and promote excellent Free Software. The mgmt project considers both Fedora and Debian to be first class platforms, and parity is a primary design goal.

One hour hacks: Remote LUKS over SSH

Mon, 25 Apr 2016 13:46:53 +0000

I have a GNU/Linux server which I mount a few LUKS encrypted drives on. I only ever interact with the server over SSH, and I never want to keep the LUKS credentials on the remote server. I don’t have anything especially sensitive on the drives, but I think it’s a good security practice to encrypt it all, if only to add noise into the system and for solidarity with those who harbour much more sensitive data.

Automatic grouping in mgmt

Wed, 30 Mar 2016 06:03:47 +0000

In this post, I’ll tell you about the recently released “automatic grouping” or “AutoGroup” feature in mgmt, a next generation configuration management prototype. If you aren’t already familiar with mgmt, I’d recommend you start by reading the introductory post, and the second post. There’s also an introductory video.

Resources in a graph

Most configuration management systems use something called a directed acyclic graph, or DAG. This is a fancy way of saying that it is a bunch of circles (vertices) which are connected with arrows (edges). The arrows must be connected to exactly two vertices, and you’re only allowed to move along each arrow in one direction (directed). Lastly, if you start at any vertex in the graph, you must never be able to return to where you started by following the arrows (acyclic). If you can, the graph is not fit for our purpose.

Automatic edges in mgmt

Mon, 14 Mar 2016 00:50:48 +0000

It’s been two months since I announced mgmt, and now it’s time to continue the story by telling you more about the design of what’s now in git master. Before I get into those details, let me quickly recap what’s happened since then.

Mgmt community recap:

I gave the first public presentation about mgmt at CfgMgmtCamp.
I repeated the talk at DevConf.cz. The video recording is available.
Felix wrote about his work cross compiling puppet code to mgmt.
Three new contributors got their feet wet in git master.
Mgmt got a lot more attention on the web...

Okay, time to tell you what’s new in mgmt!

Introducing: git tpush

Tue, 16 Feb 2016 12:15:06 +0000

On today’s issue of “one hour hacks”, I’ll show you how you can stop your git drive-by’s to git master from breaking your CI tests… Let’s continue!

The problem:

Sometimes I’ve got a shitty one-line patch that I want to push to git master. I’m usually right, and everything tests out fine, but usually isn’t always, and then I look silly while I frantically try to fix git master on a project that I maintain. Let’s use tools to hide our human flaws!

Debugging golang programs

Mon, 15 Feb 2016 16:47:57 +0000

I’ve been writing a lot of golang lately. I’ve hit painful problems in the past. Here are some debugging tips. Hopefully they help you out. I bet you don’t know #2.

#0 Use log.Printf:

This should go without saying, but I’m ashamed to say it’s what I use the most. We’ve only been C programming for 44+ years, and it’s still what is most useful!

#1 Use go run -race:

Since many problems are caused by random races, ensuring you use the built-in tools will help you catch some problems. One of these is a race detector. You can use it by adding the -race flag to the go run, go build or go test commands. In practice, it only ever caught beginner issues, and it hasn’t set off any alarms since. Maybe I need to write more test cases! Maybe you need to write more test cases!

Next generation configuration mgmt

Mon, 18 Jan 2016 00:52:10 +0000

It’s no secret to the readers of this blog that I’ve been active in the configuration management space for some time. I owe most of my knowledge to what I’ve learned while working with Puppet and from other hackers working in and around various other communities.

I’ve published, a number, of articles, in an, attempt, to push, the field, forwards, and to, share the, knowledge, that I’ve, learned, with others. I’ve spent many nights thinking about these problems, but it is not without some chagrin that I realized that the current state-of-the-art in configuration management cannot easily (or elegantly) solve all the problems for which I wish to write solutions.

Upcoming speaking

Mon, 18 Jan 2016 00:32:52 +0000

I’ve got a few upcoming speaking engagements. If you’ll be attending one of these events, come see me or any of the other excellent speakers!

Please remember to check the official schedules in case there are any changes!

I’ll be speaking at the Brussels CentOS Dojo:

Automated Infrastructure Testing with Oh-My-Vagrant ...and the CentOS CI

Time/date unconfirmed: I'll be showing some CI tricks, and showing you how the CentOS CI is the perfect CI for multi-machine test environments.

Trying out Ceph with Oh-My-Vagrant

Mon, 28 Dec 2015 14:59:23 +0000

Daniel P. Berrangé wrote about trying out a single node ceph cluster. I decided to take his article and turn it into an Oh-My-Vagrant omv.yaml file. It took me about two minutes to do so, and two hours to debug a problem caused by something I had broken on my laptop.

If you’d like to replicate his article in less than 5 minutes, pull down the omv.yaml file that I’ve just published and run omv up. Here’s the full terminal output of my session:

Matching arbitrary URL's to custom Firefox profiles

Sat, 14 Nov 2015 06:50:35 +0000

We’re constantly clicking on all sorts of different URL’s throughout the day. These clickable links appear in webpages (including in “web apps” like gmail) in mail clients like Evolution, in terminals such as GNOME-terminal, and any other GTK+ app on your GNU/Linux desktop. I wanted to perform custom actions when arbitrary URL’s are clicked, including running certain links in separate Firefox profiles. There are a bunch of different steps you have to do to get this working, but it should be easy to follow along. I’m doing all of this on Fedora 23, but it should work on other GNU/Linux environments.

Thanking Oh-My-Vagrant contributors for version 1.0.0

Tue, 18 Aug 2015 03:12:26 +0000

The Oh-My-Vagrant project became public about one year ago and at the time it was more of a fancy template than a robust project, but 188 commits (and counting) later, it has gotten surprisingly useful and mature.

james@computer:~/code/oh-my-vagrant$ git rev-list HEAD --count
188
james@computer:~/code/oh-my-vagrant$ git log $(git log --pretty=format:%H|tail -1)
commit 4faa6c89cce01c62130ef5a6d5fa0fff833da371
Author: James Shubin <james@shubin.ca>
Date:   Thu Aug 28 01:08:03 2014 -0400

    Initial commit of vagrant-puppet-docker-template...
    
    This is an attempt to prototype a default environment for
    vagrant+puppet+docker hacking. More improvements are needed for it to be
    useful, but it's probably already useful as a reference for now.

It would be easy to take most of the credit for taking the project this far, as I’ve been responsible for about 87% of the commits, but as is common, the numbers don’t tell the whole story. It is also a bug (but hopefully just an artifact) that I’ve had such a large percentage of commits. It’s quite common for a new project to start this way, but for Free Software to succeed long-term, it’s essential that the users become the contributors. Let’s try to change that going forward.

Vagrant and Oh-My-Vagrant on RHEL7

Tue, 11 Aug 2015 15:09:16 +0000

My employer keeps paying me, which I appreciate, so it’s good to spend some time to make sure RHEL7 customers get a great developer experience! So here’s how to make vagrant, vagrant-libvirt and Oh-My-Vagrant work on RHEL 7+. The same steps should work for CentOS 7+.

I’ll first paste the commands you need to run, and then I’ll explain what’s happening for those that are interested:

# run these commands, and then get hacking!
# this requires the rhel-7-server-optional-rpms repo enabled
sudo subscription-manager repos --enable rhel-7-server-optional-rpms
sudo yum install -y gcc ruby-devel libvirt-devel libvirt qemu-kvm
sudo systemctl start libvirtd.service
wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4_x86_64.rpm
sudo yum install -y vagrant_1.7.4_x86_64.rpm
vagrant plugin install vagrant-libvirt
wget https://copr.fedoraproject.org/coprs/purpleidea/vagrant-libvirt/repo/epel-7/purpleidea-vagrant-libvirt-epel-7.repo
sudo cp -a purpleidea-vagrant-libvirt-epel-7.repo /etc/yum.repos.d/
sudo yum install -y vagrant-libvirt    # noop plugin for oh-my-vagrant dependency
wget https://copr.fedoraproject.org/coprs/purpleidea/oh-my-vagrant/repo/epel-7/purpleidea-oh-my-vagrant-epel-7.repo
sudo cp -a purpleidea-oh-my-vagrant-epel-7.repo /etc/yum.repos.d/
sudo yum install -y oh-my-vagrant
. /etc/profile.d/oh-my-vagrant.sh # logout/login or source

Let’s go through it line by line.

Making an empty RPM

Tue, 11 Aug 2015 13:25:58 +0000

I am definitely not an RPM expert, in fact, I’m afraid of it, but with recent tools such as COPR, and my glorious Makefile, some aspects of it have become palatable. This post will be about a recent journey I had building the most useless RPM ever.

A video of my journey building this RPM.

Because of reasons, I wanted to satisfy an RPM dependency for a package that I wanted to install without rebuilding that RPM. As a result, I wanted to build as small an RPM as possible. This took me down a much longer path than I thought it would.

Golang parallelism issues causing “too many open files” error

Mon, 27 Jul 2015 11:39:45 +0000

I’ve been hacking in golang for a while, but I’ll admit that I didn’t get too deep into some of the language nuances until more recently. Since some of them have started to bite me, here’s a little post-mortem of one of the problems I was having.

After hacking and testing code all day, I made a seemingly innocuous change, and when running my program, I saw the following error:

2015/07/10 14:34:12 too many open files

I didn’t know what I broke, but it was obviously my fault. I reverted my recent changes, but still the error persisted. Internet searches and many painful hours of debugging ensued.

Git archive with submodules and tar magic

Thu, 23 Jul 2015 20:32:27 +0000

Git submodules are actually a very beautiful thing. You might prefer the word powerful or elegant, but that’s not the point. The downside is that they are sometimes misused, so as always, use with care. I’ve used them in projects like puppet-gluster, oh-my-vagrant, and others. If you’re not familiar with them, do a bit of reading and come back later, I’ll wait.

I recently did some work packaging Oh-My-Vagrant as RPM’s. My primary goal was to make sure the entire process was automatic, as I have no patience for manually building RPM’s. Any good packager knows that the pre-requisite for building a SRPM is a source tarball, and I wanted to build those automatically too.

Oh-My-Vagrant “Mainstream” mode and COPR RPM's

Wed, 08 Jul 2015 01:14:19 +0000

Making Oh-My-Vagrant (OMV) more developer accessible and easy to install (from a distribution package like RPM) has always been a goal, but was previously never a priority. This is all sorted out now. In this article, I’ll explain how “mainstream” mode works, and how the RPM work was done. (I promise this will be somewhat interesting!)

Prerequisites:

If you haven’t read any of the previous articles about Oh-My-Vagrant, I’d recommend you start there. Many of the articles include screencasts, and combined with the examples/ folder, this is probably the best way to learn OMV, because the documentation could use some love.

A super privileged Puppet container

Fri, 12 Jun 2015 18:06:55 +0000

In this new crazy world of containers and immutable hosts, one might still want to run previous generation software such as Puppet on a current generation Atomic host. This article will explain how you can do that, and offer some proof of concept code.

The atomic host doesn’t provide a yum or dnf command, because the software is pre-baked into a read-only /usr/ partition. To “install” (to use) additional software, it usually needs to be distributed and run as a container.

Kubernetes clusters with Oh-My-Vagrant

Sat, 02 May 2015 00:07:11 +0000

I’ve added the ability to deploy a Kubernetes cluster with Oh-My-Vagrant (omv). I’ve also built an automated developer experience so that you can test your Kubernetes powered app in minutes. If you want to redeploy a new version, or see how your app behaves during a rolling update, you can use omv to test this out in minutes! I’ve recorded a screencast (~15 min), if you’d like to see some of this in action.

Docker containers in Oh-My-Vagrant

Mon, 20 Apr 2015 04:35:43 +0000

The Oh-My-Vagrant (omv) project is an easy way to bootstrap a development environment. It is particularly useful for spinning up an arbitrary number of virtual machines in Vagrant without writing ruby code. For multi-machine container development, omv can be used to help this happen more naturally.

Oh-My-Vagrant can be very useful as a docker application development environment. I’ve made a quick (<9min) screencast demoing this topic. Please have a look:

https://dl.fedoraproject.org/pub/alt/purpleidea/screencasts/oh-my-vagrant-docker-screencast.ogv

Sharing dev environments with Oh-My-Vagrant

Wed, 08 Apr 2015 03:37:18 +0000

With Oh-My-Vagrant (omv) you can set up a dev environment in seconds. (Read the omv introduction if you’ve never used it before!) Since everything is defined in a single omv.yaml file, it is easy to share your cluster prototype with a friend! The one missing feature was associating code with this config file. This is now possible! Let me show you how it works…

In the omv.yaml file there is an extern variable. It is a list of each external repository which you’d like to include. Each element in this list is a hash of key value pairs. Currently four are supported: type, system, repository, directory.

Fancy git aliases and git cherryfetch

Mon, 16 Mar 2015 05:36:19 +0000

Here are two quick git tricks that I’ve added to my toolbox lately…

I wanted to create a git alias that takes in argv from the command, but in the middle of the command. Here’s the hack that I came up with for the [alias] section of my ~/.gitconfig:

[alias]

    # cherryfetch fetches a repo ($1) / branch ($2) and applies it rebased!
    # the && true at the end eats up the appended args
    cherryfetch = !git fetch "$1" "$2" && git cherry-pick HEAD..FETCH_HEAD && true

Explanation:

Building RHEL Vagrant Boxes with Vagrant-Builder

Mon, 23 Feb 2015 22:48:39 +0000

Vagrant is a great tool for development, but Red Hat Enterprise Linux (RHEL) customers have typically been left out, because it has been impossible to get RHEL boxes! It would be extremely elegant if hackers could quickly test and prototype their code on the same OS as they’re running in production.

Secondly, when hacking on projects that have a long initial setup phase (eg: a long rpm install) it would be excellent if hackers could roll their own modified base boxes, so that certain common operations could be re-factored out into the base image.

Introducing: Silent counter

Mon, 09 Feb 2015 07:31:58 +0000

You might want to write code that can tell how many iterations have passed since some action occurred. Alternatively, you might want to know if it’s the first time a machine has run Puppet. To do these types of things, you might wish to have a monotonically increasing counter in your Puppet manifest. Since one did not exist, I set out to build one!

The code:

If you just want to try the code, and skip the ramble, you can include common::counter into your manifest. The entire class is part of my puppet-common module:

Replying to mailing lists with Evolution

Thu, 27 Nov 2014 20:28:54 +0000

I use the Evolution mail client. It does have a few annoying bugs, but it has a plethora of great features too! Hopefully this post will inspire you to help hack on this piece of software and fix the bugs!

Mailing list etiquette:

When replying to mailing lists, it’s typically very friendly to include the email address of the person you’re replying to in the to or cc fields along with the mailing list address. This lets that person know that someone has answered their question. In some cases, if they’re not subscribed to that mailing list, (if you don’t do this), then they might not see your reply at all.

Captive web portals are considered harmful

Thu, 27 Nov 2014 00:46:06 +0000

Recently, when I tried to access http://slashdot.org/ in Firefox, I would see my browser title bar flash briefly to “AT&T GUI”, and then I would get redirected to: http://slashdot.org/cgi-bin/redirect.ha which returns slashdot’s custom error 404 page! What is going on? (Read on for answer…)

Did slashdot mess up their mod_rewrite config? (Nope, works fine in a different browser...)
Did my HTTPS everywhere extension go crazy? (Nope, still broken when disabled...)
Are my HTTP requests being MITM-ed? (Yes, probably by the NSA, but they wouldn't make this kind of mistake...)
Is my computer p0wned? (I use GNU/Linux, so probably not...)

A keyword search will show you that others are also affected by this, except that the base domain (slashdot.org) is usually different... One thing that all the links I viewed have in common: none of them seem to know what's happening.

Some background:

The switch as an ordinary GNU/Linux server

Tue, 04 Nov 2014 02:23:52 +0000

The fact that we manage the switches in our data centres differently than any other server is patently absurd, but we do so because we want to harness the power of a tiny bit of silicon which happens to be able to dramatically speed up the switching bandwidth.

beware of proprietary silicon, it's absurd!

That tiny bit of silicon is known as an ASIC, or an application specific integrated circuit, and one particularly well performing ASIC (which is present in many commercially available switches) is called the Trident.

Testing Evolution's git master and GNOME continuous

Wed, 22 Oct 2014 17:22:56 +0000

I’ve wanted a feature in Evolution for a while. It was formally requested in 2002, and it just recently got fixed in git master. I only started publicly groaning about this missing feature in 2013, and mcrha finally patched it. I tested the feature and found a small bug, mcrha patched that too, and I finally re-tested it. Now I’m blogging about this process so that you can get involved too!

Why Evolution?

Evolution supports GPG (Geary doesn't, Gmail doesn't)
Evolution has a beautiful composer (Gmail's sucks, just try to reply inline)
Evolution is Open Source and Free Software (Gmail is proprietary)
Evolution integrates with GNOME (Gmail doesn't)
Evolution has lots of fancy, mature features (Geary doesn't)
Evolution cares about your privacy (Gmail doesn't)

The feature:

I’d like to be able to select a bunch of messages and click an archive action to move them to a specific folder. Gmail popularized this idea in 2004, two years after it was proposed for Evolution. It has finally landed.

Hacking out an Openshift app

Sat, 18 Oct 2014 16:43:00 +0000

I had an itch to scratch, and I wanted to get a bit more familiar with Openshift. I had used it in the past, but it was time to have another go. The app and the code are now available. Feel free to check out:

https://pdfdoc-purpleidea.rhcloud.com/

This is a simple app that takes the URL of a markdown file on GitHub, and outputs a pandoc converted PDF. I wanted to use pandoc specifically, because it produces PDF's that were beautifully created with LaTeX. To embed a link in your upstream documentation that points to a PDF, just append the file's URL to this app's url, under a /pdf/ path. For example:

https://pdfdoc-purpleidea.rhcloud.com/pdf/https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Continuous integration for Puppet modules

Fri, 10 Oct 2014 15:34:53 +0000

I just patched puppet-gluster and puppet-ipa to bring their infrastructure up to date with the current state of affairs…

What’s new?

Better README's
Rake syntax checking (fewer oopsies)
CI (testing) with travis on git push (automatic testing for everyone)
Use of .pmtignore to ignore files from puppet module packages (finally)
Pushing modules to the forge with blacksmith (sweet!)

This last point deserves another mention. Puppetlabs created the "forge" to try to provide some sort of added value to their stewardship. Personally, I like to look for code on github instead, but nevertheless, some do use the forge. The problem is that to upload new releases, you need to click your mouse like a windows user! Someone has finally solved that problem! If you use blacksmith, a new build is just a rake push away!

Have a look at this example commit if you’re interested in seeing the plumbing.

Fixing dropbox “conflicted copy” problems

Tue, 30 Sep 2014 01:05:07 +0000

I usually avoid proprietary cloud services because of freedom, privacy and vendor lock-in concerns. In addition, there are some excellent libre (and hosted) services such as WordPress, Wikipedia and OpenShift which don’t have the above problems. Thirdly, there are every day Free Software tools such as Fedora GNU/Linux, Libreoffice, and git-annex-assistant which make my computing much more powerful. Finally, there are some hosted services that I use that don’t lock me in because I use them as push-only mirrors, and I only interact with them using Free Software tools. The two examples are GitHub and Dropbox.

Introducing: Oh My Vagrant!

Wed, 03 Sep 2014 23:19:19 +0000

If you’re a reader of my code or of this blog, it’s no secret that I hack on a lot of puppet and vagrant. Recently I’ve fooled around with a bit of docker, too. I realized that the vagrant, environments I built for puppet-gluster and puppet-ipa needed to be generalized, and they needed new features too. Therefore…

Introducing: Oh My Vagrant!

Oh My Vagrant is an attempt to provide an easy to use development environment so that you can be up and hacking quickly, and focusing on the real devops problems. The README explains my choice of project name.

Rough data density calculations

Wed, 27 Aug 2014 05:52:47 +0000

Seagate has just publicly announced 8TB HDD’s in a 3.5" form factor. I decided to do some rough calculations to understand the density a bit better…

Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since I always work in base 2, but I hate the -bi naming conventions. Seagate is most likely announcing an 8TB HDD, which is actually smaller than a true 8TiB drive. If you don’t know the difference it’s worth learning.

Hybrid management of FreeIPA types with Puppet

Thu, 24 Jul 2014 00:41:08 +0000

(Note: this hybrid management technique is being demonstrated in the puppet-ipa module for FreeIPA, but the idea could be used for other modules and scenarios too. See below for some use cases…)

The error message that puppet hackers are probably most familiar is:

Error: Duplicate declaration: Thing[/foo/bar] is already declared in file /tmp/baz.pp:2; 
cannot redeclare at /tmp/baz.pp:4 on node computer.example.com

Typically this means that there is either a bug in your code, or someone has defined something more than once. As annoying as this might be, a compile error happens for a reason: puppet detected a problem, and it is giving you a chance to fix it, without first running code that could otherwise leave your machine in an undefined state.

One minute hacks: the nautilus scripts folder

Thu, 26 Jun 2014 00:01:10 +0000

Master SDN hacker Flavio sent me some tunes. They were sitting on my desktop in a folder:

$ ls ~/Desktop/
uncopyrighted_tunes_from_flavio/

I wanted to listen them while hacking, but what was the easiest way…? I wanted to use the nautilus file browser to select which folder to play, and the totem music/video player to do the playing.

Drop a file named totem into:

~/.local/share/nautilus/scripts/

with the contents:

#!/bin/bash
# o hai from purpleidea
exec totem -- "$@"

and make it executable with:

Securely managing secrets for FreeIPA with Puppet

Fri, 06 Jun 2014 06:12:19 +0000

Configuration management is an essential part of securing your infrastructure because it can make sure that it is set up correctly. It is essential that configuration management only enhance security, and not weaken it. Unfortunately, the status-quo of secret management in puppet is pretty poor.

In the worst (and most common) case, plain text passwords are found in manifests. If the module author tried harder, sometimes these password strings are pre-hashed (and sometimes salted) and fed directly into the consumer. (This isn’t always possible without modifying the software you’re managing.)

Hiera data in modules and OS independent puppet

Wed, 04 Jun 2014 03:47:23 +0000

Earlier this year, R.I.Pienaar released his brilliant data in modules hack, a few months ago, I got the chance to start implementing it in Puppet-Gluster, and today I have found the time to blog about it.

What is it?

R.I.’s hack lets you store hiera data inside a puppet module. This can have many uses including letting you throw out the nested mess that is commonly params.pp, and replace it with something file based that is elegant and hierarchical. For my use case, I’m using it to build OS independent puppet modules, without storing this data as code. The secondary win is that porting your module to a new GNU/Linux distribution or version could be as simple as adding a YAML file.

Restarting GNOME shell via SSH

Thu, 29 May 2014 02:18:35 +0000

When GNOME shell breaks, you get to keep both pieces. The nice thing about shell failures in GNOME 3, is that if you’re able to do a restart, the active windows are mostly not disturbed. The common way to do this is to type ALT-F2, r, «em>ENTER>.

Unfortunately, you can’t always type that in if your shell is very borked. If you are lucky enough to have SSH access, and another machine, you can log in remotely and run this script:

Vagrant on Fedora with libvirt (reprise)

Tue, 13 May 2014 18:58:06 +0000

Vagrant has become the de facto tool for devops. Faster iterations, clean environments, and less overhead. This isn’t an article about why you should use Vagrant. This is an article about how to get up and running with Vagrant on Fedora with libvirt easily!

Background:

This article is an update of my original Vagrant on Fedora with libvirt article. There is still lots of good information in that article, but this one should be easier to follow and uses updated versions of Vagrant and vagrant-libvirt.

Keeping git submodules in sync with your branches

Tue, 06 May 2014 19:34:13 +0000

This is a quick trick for making working with git submodules more magic.

One day you might find that using git submodules is needed for your project. It’s probably not necessary for everyday hacking, but if you’re glue-ing things together, it can be quite useful. Puppet-Gluster uses this technique to easily include all the dependencies needed for a Puppet-Gluster+Vagrant automatic deployment.

If you’re a good hacker, you develop things in separate feature branches. Example:

Working at RedHat

Wed, 02 Apr 2014 14:27:35 +0000

So this happened:

James just James at RedHat headquarters in North Carolina wearing his new red hat.

RedHat made me an offer, and I am happy to say that I have just started this week!

I am proud to have joined a company that employs many of the worlds foremost, highly professional and clever hackers. It is indubitably the best Free Software [1] / Open Source company out there, and they ship some of the greatest and most elegant software available.

Puppet-Gluster now available as RPM

Thu, 27 Mar 2014 20:32:41 +0000

I’ve been afraid of RPM and package maintaining [1] for years, but thanks to Kaleb Keithley, I have finally made some RPM’s that weren’t generated from a high level tool. Now that I have the boilerplate done, it’s a relatively painless process!

In case you don’t know kkeithley, he is a wizard [2] who happens to also be especially cool and hardworking. If you meet him, be sure to buy him a $BEVERAGE. </plug>

A photo of kkeithley after he (temporarily) transformed himself into a wizard penguin.

The full source of my changes is available in git.

Introducing Puppet Exec['again']

Mon, 24 Mar 2014 18:37:23 +0000

Puppet is missing a number of much-needed features. That’s the bad news. The good news is that I’ve been able to write some of these as modules that don’t need to change the Puppet core! This is an article about one of these features.

Posit: It’s not possible to apply all of your Puppet manifests in a single run.

I believe that this holds true for the current implementation of Puppet. Most manifests can, do and should apply completely in a single run. If your Puppet run takes more than one run to converge, then chances are that you’re doing something wrong.

Preserving your working directory in gnome-terminal

Thu, 20 Mar 2014 18:01:24 +0000

I use gnome-terminal for most of my hacking. In fact, I use it so much, that I’ll often have multiple tabs open for a particular project. Here’s my workflow:

Control+Alt+t (My shortcut to open a new gnome-terminal window.)
cd ~/code/some_cool_hack/ # directory of some cool hack
Control-Shift-t (Shortcut to open a new gnome-terminal tab.)
Hack, hack, hack...

The problem is that the new tab that I've created will have a $PWD of ~, instead of keeping the $PWD of ~/code/some_cool_hack/, which is the project I'm working on!

The solution is to add:

Speaking at SCALE today!

Fri, 21 Feb 2014 13:54:47 +0000

I’ll be giving a talk at SCALE today about automatically deploying GlusterFS with Puppet-Gluster and Vagrant. I’ll be giving some live demos, and this will cover some of the material from:

Automatically deploying GlusterFS with Puppet-Gluster + Vagrant!

and it will contain excerpts from:

Screencasts of Puppet-Gluster + Vagrant

I’ll also be talking about some new upcoming features, and am happy to answer all of your questions!

The talk will be part of Infrastructure.next and is starting around 1:30 or 2pm in the Century AB room.

Building a snow shelter

Mon, 17 Feb 2014 18:23:27 +0000

To give you a break from the usual GNU/Linux/DevOps/Puppet/GlusterFS drab, I’ve decided to have a go at writing a different kind of technical article. This article will show you how to build the traditional Canadian snow dwelling known as a quinzee. If you will be travelling to Canada, I recommended that you read through this article ahead of time, so that you don’t offend your host by being unfamiliar with their traditional living accommodations.

Scathing review of the Lenovo X240

Sun, 02 Feb 2014 15:15:10 +0000

I’m using a Lenovo X201 with 8GiB of RAM. Apart from some minor issues, I’ve been very satisfied with this laptop. It’s over four years old, and so I decided to see what’s available on the horizon. I did not buy an X240 because of the following reasons:

The X240 has only one slot for RAM and thus supports a maximum of 8GiB.

I think it’s pretty ridiculous for any successor to the X230 to support less RAM. In and of itself this is a deal breaker! My X201 has 8GiB, and I wish it had more! It doesn’t make sense to settle for less on new kit. In other words, Lenovo is stating that 8GiB ought to be enough for anybody.

Show the exit status in your $PS1

Wed, 29 Jan 2014 19:43:44 +0000

As an update to my earlier article, a friend gave me an idea of how to make my $PS1 even better… First, the relevant part of my ~/.bashrc:

ps1_prompt() {
	local ps1_exit=$?

	if [ $ps1_exit -eq 0 ]; then
		#ps1_status=`echo -e "\[&#092;&#048;33[32m\]"'\$'"\[&#092;&#048;33[0m\]"`
		ps1_status='\$'
	else
		ps1_status=`echo -e "\[&#092;&#048;33[1;31m\]"'\$'"\[&#092;&#048;33[0m\]"`

	fi

	ps1_git=''
	if [ "$(__git_ps1 %s)" != '' -a "$(__git_ps1 %s)" != 'master' ]; then
		ps1_git=" (\[&#092;&#048;33[32m\]"$(__git_ps1 "%s")"\[&#092;&#048;33[0m\])"
	fi

	PS1="${debian_chroot:+($debian_chroot)}\u@\h:\[&#092;&#048;33[01;34m\]\w\[&#092;&#048;33[00m\]${ps1_git}${ps1_status} "
}

# preserve earlier PROMPT_COMMAND entries...
PROMPT_COMMAND="ps1_prompt;$PROMPT_COMMAND"

If you haven’t figured it out, the magic is that the trailing $ prompt gets coloured in red when the previous command exited with a non-zero value. Example:

Screencasts of Puppet-Gluster + Vagrant

Mon, 27 Jan 2014 16:13:10 +0000

I decided to record some screencasts to show how easy it is to deploy GlusterFS using Puppet-Gluster+Vagrant. You can follow along even if you don’t know anything about Puppet or Vagrant. The hardest part of this process was producing the actual videos!

If recommend first reading my earlier articles if you’re planning on following along:

Without any further delay, here are the screencasts:

Part 1: Intro, and provisioning of the Puppet server.

Building base images for Vagrant with a Makefile

Mon, 20 Jan 2014 11:46:31 +0000

I needed a base image “box” for my Puppet-Gluster+Vagrant work. It would have been great if good boxes already existed, and even better if it were easy to build my own. As it turns out, I wasn’t able to satisfy either of these conditions, so I’ve had to build one myself! I’ve published all of my code, so that you can use these techniques and tools too!

Testing GlusterFS during “Glusterfest”

Thu, 16 Jan 2014 20:28:17 +0000

The GlusterFS community is having a “test day”. Puppet-Gluster+Vagrant is a great tool to help with this, and it has now been patched to support alpha, beta, qa, and rc releases! Because it was built so well (*cough*, shameless plug), it only took one patch.

Okay, first make sure that your Puppet-Gluster+Vagrant setup is working properly. I have only tested this on Fedora 20. Please read:

Automatically deploying GlusterFS with Puppet-Gluster + Vagrant!

Wed, 08 Jan 2014 23:00:22 +0000

Puppet-Gluster was always about automating the deployment of GlusterFS. Getting your own Puppet server and the associated infrastructure running was never included “out of the box”. Today, it is! (This is big news!)

I’ve used Vagrant to automatically build these GlusterFS clusters. I’ve tested this with Fedora 20, and vagrant-libvirt. This won’t work with Fedora 19 because of bz#876541. I recommend first reading my earlier articles for Vagrant and Fedora:

Once you're comfortable with the material in the above articles, we can continue...

The short answer:

Vagrant clustered SSH and ‘screen’

Thu, 02 Jan 2014 00:40:49 +0000

Some fun updates for vagrant hackers… I wanted to use the venerable clustered SSH (cssh) and screen with vagrant. I decided to expand on my vsftp script. First read:

Vagrant on Fedora with libvirt

and

Vagrant vsftp and other tricks

to get up to speed on the background information.

Vagrant screen:

First, a simple screen hack… I often use my vssh alias to quickly ssh into a machine, but I don’t want to have to waste time with sudo-ing to root and then running screen each time. Enter vscreen:

Vagrant vsftp and other tricks

Sat, 21 Dec 2013 11:58:49 +0000

As I previously wrote, I’ve been busy with Vagrant on Fedora with libvirt, and have even been submitting, patches and issues! (This “closed” issue needs solving!) Here are some of the tricks that I’ve used while hacking away.

Default provider:

I should have mentioned this in my earlier article but I forgot: If you’re always using the same provider, you might want to set it as the default. In my case I’m using vagrant-libvirt. To do so, add the following to your .bashrc:

Vagrant on Fedora with libvirt

Mon, 09 Dec 2013 20:14:51 +0000

Apparently lots of people are using Vagrant these days, so I figured I’d try it out. I wanted to get it working on Fedora, and without Virtualbox. This is an intro article on Vagrant, and what I’ve done. I did this on Fedora 19. Feel free to suggest improvements.

Intro:

Vagrant is a tool that easily provisions virtual machines, and then kicks off a configuration management deployment like Puppet. It’s often used for development. I’m planning on using it to test some Puppet-Gluster managed GlusterFS clusters.

Now syndicated on “Planet Fedora”

Mon, 09 Dec 2013 19:18:56 +0000

I’m now syndicated on the Fedora Project planet. If you haven’t read through my blog yet, let me introduce myself, I’m James, and I write The Technical Blog of James.

I’m a sysadmin, DevOps/Puppet hacker, I.T./network architect and physiologist. Hi! I run Fedora as my primary desktop, but I also use it for servers, particularly for development before future versions of RHEL and CentOS release.

I’m most well-known for Puppet-Gluster, but I’ve also written a decent Puppet-IPA (FreeIPA) module. I’m currently working on porting Puppet-Gluster to Fedora and looking forward to the upcoming release of Fedora 20.

Advanced recursion and memoization in Puppet

Wed, 27 Nov 2013 00:03:15 +0000

As a follow-up to my original article on recursion in Puppet, and in my attempt to Push Puppet (to its limit), I’ll now attempt some more advanced recursion techniques in Puppet.

In my original recursion example, the type does recurse, but the callee cannot return any value to the caller because it is a type, and not strictly a function. This limitation immediately limits the usefulness of this technique, but I’ll try to press on! Let’s try to write a Fibonacci series function in native Puppet.

Documentation for Puppet-Gluster

Wed, 20 Nov 2013 06:36:19 +0000

Ironically, one of the reasons that I started writing Puppet code, was so that I could spend more time designing and building, and less time writing documentation. I suppose I’m a victim of my success, because Puppet-Gluster has grown large enough to warrant its own documentation.

So I gave in, and put together some documentation.

It’s available as markdown, and, also as a pdf. As before, there is an examples/ directory which you might want to use as additional reference.

Iteration in Puppet

Sun, 17 Nov 2013 04:25:01 +0000

People often ask how to do iteration in Puppet. Most Puppet users have a background in imperative programming, and are already very familiar with for loops. Puppet is sometimes confusing at first, because it is actually (or technically, contains) a declarative, domain-specific language. In general, DSL’s aren’t always Turing complete, nor do they need to support loops, but this doesn’t mean you can’t iterate.

Until recently, Puppet didn’t have an explicit looping construct, and it is quite possible to build complex modules without using this new construct. There are even some who believe that the language shouldn’t even contain this feature. I’ll abstain from that debate for now, but instead, I would like to show you some iteration techniques that you can use to get your desired result.

Pushing Puppet at Puppet Camp DC, LISA 2013

Tue, 05 Nov 2013 15:00:19 +0000

Hi there,

I hope you enjoyed my “Pushing Puppet (to its limit)” talk and demos from Puppet Camp D.C., LISA 2013. As requested, I’ve posted the code and slides.

Here is the code: https://github.com/purpleidea/puppet-pushing

This module will require three modules as dependencies. The dependencies are:

My Puppet-Common module https://github.com/purpleidea/puppet-common
My Puppet-Runonce module https://github.com/purpleidea/puppet-runonce
My Puppet-FSM module https://github.com/purpleidea/puppet-fsm

Each example doesn't require all the dependencies, so if you're only interested in the FSM, you only need that module.

Here are the slides: https://github.com/purpleidea/puppet-pushing/blob/master/talks/pushing-puppet.pdf

Gluster Community Day, LISA 2013, Monday

Tue, 05 Nov 2013 01:30:05 +0000

I’m here at LISA 2013 at the Gluster Community Day. I’ve been asked by Joe Brockmeier to give a little recap about what’s been going on. So here it is!

Wesley Duffee-Braun started off with a nice overview talk about GlusterFS. The great thing about his talk was that he gave a live demo, running on virtual machines, on his laptop. If you’re a new GlusterFS user, this is good exposure to help you get started. He also has the nicest slides I’ve ever seen. Someone needs to send me the .odp template!

Speaking at LISA 2013 about Puppet and GlusterFS

Sun, 03 Nov 2013 19:40:08 +0000

I’m speaking at LISA 2013, the “Large Installation System Administration” conference. This conference runs all week in Washington. I’ll be giving two talks during the week, and attending at least one BOF.

My first talk is on Monday during the Gluster Community Day. I’ll be speaking about puppet-gluster, and giving a live demo. I’ll be showing some new features too. If you’d like to talk more about puppet-gluster, or want to attend the talk, give me a shout, or sign up at the above Gluster Community Day link.

Easier strace of scripts with pidof -x

Sat, 26 Oct 2013 16:02:12 +0000

Here’s a one minute read, about a trick which I discovered today:

When running an strace, it’s common to do something like:

strace -p<pid>

Smarter hackers know that they can use some bash magic and do:

strace -p`pidof <process name>`

However, if you’re tracing a script named foo.py, this won’t work because the real process is the script’s interpreter, and pidof python, might return other unrelated python scripts.

strace -p`pidof foo.py` # won't work
[failure]
[user sifting through ps output for real pid]
[computer explodes]

The trick is to use the -x flag of pidof. This will let you pass in your script’s name, and pidof will take care of the rest:

first release of puppet-shorewall

Mon, 21 Oct 2013 00:07:06 +0000

Oh, hi there.

In case you’re interested, I’ve just made a first release of my puppet-shorewall module. This isn’t meant as an exhaustive shorewall module, but it does provide most of the usual functionality that most users need.

In particular, it’s the module dependency that I use for many of my other puppet modules that provide firewalling. This is probably where you’re most likely to consume it.

In general most modules just implement shorewall::rule, so if you really don’t want to use this code, you can implement that signature yourself, or not use automatic firewalling. The shorewall::rule type has two main signatures, so have a look at the source, or a simple example if you want to get more familiar with the specifics. Using this module is highly recommended, specifically with puppet-gluster.

Desktop Notifications for Irssi in Screen through SSH in Gnome Terminal

Fri, 18 Oct 2013 06:00:49 +0000

I’m usually on IRC, but I don’t often notice incoming pings until after the fact. I had to both write, and modify various scripts to get what I wanted, but now it’s all done, and you can benefit from my hacking by following along…

The Setup

Laptop -> Gnome-Terminal -> SSH -> Screen -> Irssi

This way, I’m connected to IRC, even when my laptop isn’t. I run irssi in a screen session on an SSH server that I manage, and I use gnome-terminal on my laptop. If you don’t understand this setup, then you’ll need to get more comfortable with these tools first.

GNOME Montreal Summit

Tue, 15 Oct 2013 16:52:29 +0000

This October 12th to 14th Montreal hosted the GNOME ~~boston~~ summit. Many thanks to Canonical for sponsoring breakfast, Savoir Faire Linux for hosting a great 6 à 10 with fancy snacks, and RedHat for sponsoring a pool night. What follows is some technical commentary about stuff that went on.

JHBuild

JHBuild is a tool to make it easy to download/clone (from git) and compile all the GNOME modules and applications. It was easy to get going. I (mostly) followed the steps listed on the JHBuild HowDoI wiki page. I kept my .jhbuildrc in my home directory instead of ~/.config/. On my Fedora 19 machine, I found certain unlisted dependencies were missing to build everything. You can figure these out yourself when your builds fail, or just run the following command to install them beforehand:

Show current git branch in PS1 when branch is not master

Thu, 10 Oct 2013 22:16:28 +0000

Short post, long command…

I’ve decided to start showing the current git branch in my PS1. However, since I don’t want to know when I’m on master, I had to write a new PS1 that I haven’t yet seen anywhere. Add the following to your .bashrc:

PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
if [ -e /usr/share/git-core/contrib/completion/git-prompt.sh ]; then
    . /usr/share/git-core/contrib/completion/git-prompt.sh
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w$([ "$(__git_ps1 %s)" != "" -a "$(__git_ps1 %s)" != "master" ] && (echo -e " (\[33[32m\]"$(__git_ps1 "%s")"\[33[0m\])") || echo "")\$ '
fi

Finite state machines in puppet

Sat, 28 Sep 2013 06:09:08 +0000

In my attempt to push puppet to its limits, (for no particular reason), to develop more powerful puppet modules, to build in a distributed lock manager, and to be more dynamic, I’m now attempting to build a Finite State Machine (FSM) in puppet.

No Ads!

Thu, 26 Sep 2013 23:18:34 +0000

I just shelled out to wordpress.com to buy the No Ads upgrade. I think they’re good people. I hope you all appreciate it. Let me know if you do.

Happy hacking,

James

Installing missing GNOME games

Thu, 26 Sep 2013 00:43:50 +0000

I just realized that my Fedora 19 installation didn’t have any of the GNOME games installed by default any more. I guess there’s no love for nibbles. Here’s a quick one-liner to get them all back:

$ sudo yum search game | grep gnome | awk '{print $1}' | xargs sudo yum install -y
Loaded plugins: etckeeper, langpacks, refresh-packagekit
Package gnome-nibbles-3.8.0-2.fc19.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package gnome-chess.x86_64 0:3.8.3-2.fc19 will be installed
--> Processing Dependency: gnuchess for package: gnome-chess-3.8.3-2.fc19.x86_64
---> Package gnome-hearts.x86_64 0:0.3-12.fc19 will be installed
---> Package gnome-klotski.x86_64 0:3.8.2-2.fc19 will be installed
---> Package gnome-mahjongg.x86_64 0:3.8.0-3.fc19 will be installed
---> Package gnome-mines.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-robots.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-sudoku.noarch 1:3.8.1-3.fc19 will be installed
---> Package gnome-tetravex.x86_64 0:3.8.1-2.fc19 will be installed
--> Running transaction check
---> Package gnuchess.x86_64 0:6.0.3-1.fc19 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch          Version                Repository     Size
================================================================================
Installing:
 gnome-chess           x86_64        3.8.3-2.fc19           fedora        2.6 M
 gnome-hearts          x86_64        0.3-12.fc19            fedora        365 k
 gnome-klotski         x86_64        3.8.2-2.fc19           fedora        1.2 M
 gnome-mahjongg        x86_64        3.8.0-3.fc19           fedora        3.8 M
 gnome-mines           x86_64        3.8.1-3.fc19           fedora        2.7 M
 gnome-robots          x86_64        3.8.1-3.fc19           fedora        1.4 M
 gnome-sudoku          noarch        1:3.8.1-3.fc19         fedora        2.5 M
 gnome-tetravex        x86_64        3.8.1-2.fc19           fedora        1.7 M
Installing for dependencies:
 gnuchess              x86_64        6.0.3-1.fc19           fedora        2.6 M

Transaction Summary
================================================================================
Install  8 Packages (+1 Dependent package)

Total download size: 19 M
Installed size: 54 M
Downloading packages:
(1/9): gnome-chess-3.8.3-2.fc19.x86_64.rpm                 | 2.6 MB   00:04     
(2/9): gnome-mines-3.8.1-3.fc19.x86_64.rpm                 | 2.7 MB   00:05     
(3/9): gnome-klotski-3.8.2-2.fc19.x86_64.rpm               | 1.2 MB   00:05     
(4/9): gnome-robots-3.8.1-3.fc19.x86_64.rpm                | 1.4 MB   00:01     
(5/9): gnome-tetravex-3.8.1-2.fc19.x86_64.rpm                                                                                          | 1.7 MB  00:00:03     
(6/9): gnome-hearts-0.3-12.fc19.x86_64.rpm                                                                                             | 365 kB  00:00:10     
(7/9): gnuchess-6.0.3-1.fc19.x86_64.rpm                                                                                                | 2.6 MB  00:00:04     
(8/9): gnome-mahjongg-3.8.0-3.fc19.x86_64.rpm                                                                                          | 3.8 MB  00:00:11     
(9/9): gnome-sudoku-3.8.1-3.fc19.noarch.rpm                                                                                            | 2.5 MB  00:00:06     
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         1.5 MB/s |  19 MB     00:12     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
etckeeper: pre transaction commit
  Installing : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               1/9 
  Installing : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            2/9 
  Installing : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          3/9 
  Installing : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         4/9 
  Installing : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            5/9 
  Installing : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Installing : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            7/9 
  Installing : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         8/9 
  Installing : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           9/9 
etckeeper: post transaction commit
  Verifying  : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            1/9 
  Verifying  : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               2/9 
  Verifying  : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           3/9 
  Verifying  : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         4/9 
  Verifying  : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            5/9 
  Verifying  : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Verifying  : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            7/9 
  Verifying  : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         8/9 
  Verifying  : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          9/9 

Installed:
  gnome-chess.x86_64 0:3.8.3-2.fc19    gnome-hearts.x86_64 0:0.3-12.fc19     gnome-klotski.x86_64 0:3.8.2-2.fc19    gnome-mahjongg.x86_64 0:3.8.0-3.fc19   
  gnome-mines.x86_64 0:3.8.1-3.fc19    gnome-robots.x86_64 0:3.8.1-3.fc19    gnome-sudoku.noarch 1:3.8.1-3.fc19     gnome-tetravex.x86_64 0:3.8.1-2.fc19   

Dependency Installed:
  gnuchess.x86_64 0:6.0.3-1.fc19                                                                                                                              

Complete!
$

This isn’t a master hack, but it’s good to think about using command line magic to do your day-to-day tasks.

Bittorent sync for repository mirroring

Mon, 23 Sep 2013 02:06:28 +0000

Theron Conrey writes about using:

BitTorrent Sync as Geo-Replication for Storage

We got a chance to talk about this idea at Linuxcon. I'm not entirely convinced there aren't some problem edge cases with this solution, but I think it will be hard to tell as long as the BitTorrent sync library is proprietary. I did come up with a special case of Theron's idea that I believe could work well.

The special case uses the optimization that the synchronization (or file transferring) is unidirectional. This avoids any coherency complications involved if both sides were to write to the same file. Combined with the BitTorrent protocol, this does what normal torrent usage does, except with BitTorrent sync, we’re looking at a folder full of files.

Gluster Community Day, Thursday

Fri, 20 Sep 2013 16:23:12 +0000

I’m here in New Orleans hacking up a storm and getting to meet fellow gluster users IRL. John Mark Walker started off with a great “State of the GlusterFS union” style talk.

Today Louis (semiosis) gave a great talk about running glusterfs on amazon. It was highly pragmatic and he explained how he chose the number of bricks per host. The talk will be posted online shortly.

Marco Ceppi from Canonical gave a talk about juju and gluster. I haven’t had much time to look at juju, so it was good exposure. Marco’s gluster charm suffers from a lack of high availability peering, but I’m sure that is easily solved, and it isn’t a big issue. I had the same issue when working on puppet-gluster. I’ve written an article about how I solved this problem. I think it’s the most elegant solution, but if anyone has a better idea, please let me know. The solutions I used for puppet, can be applied to juju too. Marco and I talked about porting puppet-gluster to ubuntu. We also talked about using puppet inside of juju, with a puppetmaster, but we’re not sure how useful that would be beyond pure hack value.

Linuxcon day three, Wednesday

Wed, 18 Sep 2013 23:59:28 +0000

After hacking away on Monday and Tuesday and meeting fellow nerds IRL, I’ve landed even more changes to puppet-gluster. My git master branch now sits at 47 commits.

$ git clone https://github.com/purpleidea/puppet-gluster.git
Cloning into 'puppet-gluster'...
remote: Counting objects: 317, done.
remote: Compressing objects: 100% (144/144), done.
remote: Total 317 (delta 187), reused 275 (delta 148)
Receiving objects: 100% (317/317), 82.17 KiB | 12.00 KiB/s, done.
Resolving deltas: 100% (187/187), done.
$ cd puppet-gluster/
$ git log | grep '^commit' | wc -l
47
$ git log | head
commit <a href="https://github.com/purpleidea/puppet-gluster/commit/fa3fd2eb4bab499031274e0918a40e7a99fe0086">fa3fd2eb4bab499031274e0918a40e7a99fe0086</a>
Author: James Shubin <<em>hidden</em>>
Date:   Wed Sep 18 17:53:13 2013 -0400

    Added fancy volume creation.
    
    This moves the command into a separate file. This also adds temporary
    saving of stdout and stderr to /tmp for easy debugging of command
    output.

As you can see above, volume creation is now “fancier” and more robust. In case things go wrong, it’s easy to get fast access to gluster command line output (saved in /tmp/), and the volume creation commands are individually stored in your puppet-gluster working directory. Usually this is /var/lib/puppet/tmp/gluster/, and each volume creation command is in the volume subdirectory.

Linuxcon day two, Tuesday

Wed, 18 Sep 2013 23:05:05 +0000

Continuing on from yesterday, I’ve met even more interesting people. I chatted with Dianne Mueller about some interesting ideas for gluster+openshift. More to come on that front soon. Hung out with Jono Bacon and talked a bit about puppet-gluster on Ubuntu. If there is interest in the community for this, please let me know. Thanks to John Mark Walker and RedHat for sponsoring me and introducing me to many of these folks. Hello to all the others that I didn’t mention.

Linuxcon day one, Monday

Wed, 18 Sep 2013 22:32:16 +0000

I’m here in New Orleans at Linux Con, hacking on puppet-gluster and talking to lots of interesting folks. I’ve met gluster hacker Theron Conrey, and my host John Mark Walker, Fedora and Raspberry Pi experts Spot and Ruth Suehle, and many others too.

The hotel is very nice. The bathroom sink has two taps of course, but both of them are hot. The New Orleans heat is probably the cause of this.

I’m hacking at full speed to get some new features and testing in before my talk on Thursday. I’ve been reworking the simple firewall support in my puppet module. For those that want automated and correct firewall configuration, expect some improvements soon.

New puppet-gluster features before Linuxcon

Sun, 08 Sep 2013 04:51:14 +0000

Hey there,

I’ve done a bit of puppet-gluster hacking lately to try to squeeze some extra features and testing in before Linuxcon. Here’s a short list:

SELinux fixes to keep Dan Walsh happy :)
Ping and status checks before volume creation. Now puppet-gluster will be less noisy about failures or missing executions that are due to the necessary incremental nature of puppet-gluster runs. You'll need multiple puppet runs to get a complete setup, so don't let puppet complain part way through. This is due to the design of puppet.
Simple filesystem paths for bricks. This is very useful for prototyping with virtual machines.
Automatic UUID assignment for hosts. This requires exported resources.
Updated examples
And more...

If there are features or bugs that you'd like to see added or removed (respectively) please let me know ASAP so that I can try to get something ready for you before my Linuxcon talk. I also don't have any hardware RAID or physical hardware to test external storage partitions (bricks) on. If you have any that you can donate or let me hack on for a while, there are some features I'd like to test. Contact me!

I’ve got a few more things queued up too, but you’ll have to wait and see. Until then,

Puppet-Gluster and me at Linuxcon

Mon, 02 Sep 2013 05:50:23 +0000

John Mark Walker, (from Redhat) has been kind enough to invite me to speak at the Linuxcon Gluster Workshop in New Orleans. I’ll be speaking about puppet-gluster, giving demos, and hopefully showing off some new features. I’m also looking forward to meeting up with gluster expert Joe Julian.

If there are features that puppet-gluster is missing, or you have a use case that I haven’t covered, please let me know, and I’ll try to work on it for you ahead of the conference. If you want to meet up for some puppet-gluster help, or to hack on code, I’ll be around from the 16th to the 20th of September. My talk is on the 19th.

Finding YAML errors in puppet

Sun, 25 Aug 2013 04:31:49 +0000

I love tabs, they’re so much easier to work with, but YAML doesn’t like them. I’m constantly adding them in accidentally, and puppet’s error message is a bit cryptic:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: malformed format string - %S at /etc/puppet/manifests/foo.pp:18 on node bar.example.com

This happens during a puppet run, which in my case loads up YAML files. The tricky part was that the error wasn’t at all related to the foo.pp file, it just happened to be the first time hiera was run. So where’s the real error?

Upgrading from Fedora 18 to Fedora 19

Sun, 04 Aug 2013 05:02:18 +0000

It was time to take the plunge and upgrade from Fedora 18 to Fedora 19. Fedora 18 was one of the worst releases ever, so I figured it could only get better. I ran my backups as usual, however this time I didn’t seem to need them, the upgrade process went off without a hitch! I used the fedup-cli process over the network. I always run these things inside of screen.

a puppet-ipa user type and a new difference engine

Tue, 09 Jul 2013 06:41:54 +0000

A simple hack to add a user type to my puppet-ipa module turned out to cause quite a stir. I’ve just pushed these changes out for your testing:

3 files changed, 1401 insertions(+), 215 deletions(-)

You should now have a highly capable user type, along with some quick examples.

I’ve also done a rewrite of the difference engine, so that it is cleaner and more robust. It now uses function decorators and individual function comparators to help wrangle the data into easily comparable forms. This should make adding future types easier, and less error prone. If you’re not comfortable with ruby, that’s okay, because it’s written in python!

Fresh releases! puppet-ipa, puppet-nfs, puppet-gluster

Sun, 23 Jun 2013 09:22:32 +0000

I’ve been a little slow in making release announcements, so here’s some news:

I’ve just released the third stage of my puppet-ipa module. At the moment it now supports installation, managing of hosts, and managing of services. It integrates with my puppet-nfs module to allow you to easily setup and run an NFSv4 kerberized server and client.

While we’re at it, that’s some more news: I’ve just released a puppet-nfs module to make your /etc/exports management easier. It’s designed to manage other security types, or even to work without kerberos or any authentication at all, but I haven’t tested those.

Playing with FreeIPA and puppet

Tue, 18 Jun 2013 12:36:54 +0000

So I just rolled a new vm to hack around with FreeIPA. Here are some things that I’ve come across so far. I was planning on configuring LDAP, and Kerberos manually, but the included webui looks like a lovely tool to have for the data entry, user administrator type who likes to click on things. Let’s explore…

/etc/hosts:

FreeIPA is choosy about how your /etc/hosts is formatted. It requires an entry that has a particular order, that is:

puppet lsi hardware raid module

Mon, 17 Jun 2013 18:11:02 +0000

In response to some discussion in the gluster community, I am releasing my puppet-lsi module. It’s quite simple, but it is very useful for rebuilding machines. It could do a lot more, but I wanted to depend on the proprietary LSI tools as little as possible. Running megacli with puppet would be a very doable hack, but I’m not sure enough devops out there who would use that feature.

Usage is straightforward if you like the sensible defaults:

Collecting duplicate resources in puppet

Tue, 04 Jun 2013 23:39:08 +0000

I could probably write a long design article explaining why identical duplicate resources should be allowed [1] in puppet. If puppet is going to survive in the long-term, they will have to build in this feature. In the short-term, I will have to hack around deficiency. As luck would have it, Mr. Bode has already written part one of the hack: ensure_resource.

Why?

Suppose you have a given infrastructure with N vaguely identical nodes. N could equal 2 for a dual primary or active-passive cluster, or N could be greater than 2 for a more elaborate N-ary cluster. It is sufficient to say, that each of those N nodes might export an identical puppet resource which one (or many) clients might need to collect, to operate correctly. It’s important that each node export this, so that there is no single point of failure if one or more of the cluster nodes goes missing.

puppet matlab module

Tue, 04 Jun 2013 02:46:27 +0000

I don’t like repetitive work, and installing matlab counts doubly as so. Once I figured out the correct steps, I automated it with a puppet module. The downside is that the install takes a while because puppet needs to copy the iso locally. This is okay because I can be busy doing something else while this is happening.

Using the module is quite easy:

matlab::install { 'R2011a':
    iso => 'puppet://files/matlab/MATHWORKS_R2011A.iso',
    licensekey => '#####-#####-#####-#####',    # provide your own here
    licensefile => 'puppet:///files/matlab/license.lic',    # get your own!
    licenseagree => true,    # setting this to true 'acknowledges' their (C)
    prefix => '/usr/local',
}

You might notice that this supports installing multiple releases on the same machine. You will have to provide your own license key and license file.

puppet rsnapshot module

Tue, 04 Jun 2013 02:29:35 +0000

Today I am releasing a puppet module for rsnapshot. The nice feature of this module, is that it lets you configure multiple different instances of rsnapshot, so that they could all run in parallel. Rsnapshot doesn’t support this directly, so this puppet module does the heavy lifting of separating out and managing each instance.

https://github.com/purpleidea/puppet-rsnapshot

The examples/ directory should give you a hint on how to use it. For everything else, have a look at the code, or feel free to leave me a comment. I hope you find it useful!

Forcing firefox to remember passwords

Fri, 17 May 2013 07:39:06 +0000

There are a handful of websites out there that decide that they know better than your browser and tell it to not offer to save passwords. They do this by setting a form autocomplete attribute to off.

Since we already agree that HTML and the web are a terrible idea, hopefully we can find a way to hack around this. It turns out that I didn’t have to, because many others have solved this hack before me. The cleanest version I found is here: http://www.howtogeek.com/62980/how-to-force-your-browser-to-remember-passwords/

Overriding attributes of collected exported resources

Tue, 14 May 2013 01:41:23 +0000

This post is about a particularly elegant (and crucial) feature in puppet exported resources: attribute overriding. If you’re not already familiar with exported resources, you should start there, as they are the killer feature that makes configuration management with puppet awesome. (I haven’t found any explicit docs about this feature either, so feel free to comment if you know where they’re hidden.)

Setup: I’ve got a virtual machine which exports a resource to N different nodes. I’d like to define the resource with just one exported (@@) definition on my virtual machine.

Mothers day hacks

Sun, 12 May 2013 07:32:45 +0000

Firstly Happy Mother’s day to my mother.

Google is, as usual, busily releasing doodles. Today, the doodle takes you through a Rube Goldberg -esque sequence, giving you four decisions to make along the way. Each decision gives you one of three different choices, and at the end, a unique drawing is displayed. I expect:

3 * 3 * 3 * 3 = 81

different permutations. At the end of the process, you can print your image. I got directed to:

Upgrading to centos 6.4 with shorewall onboard

Thu, 25 Apr 2013 04:10:10 +0000

In case you upgrade your CentOS 6.x box to version 6.4, the shorewall service might complain. With a scary message:

ERROR: Your kernel/iptables do not include state match support.
No version of Shorewall will run on this system

This is selinux at work, and the problem can easily be solved by running:

# restorecon -Rv /sbin

Thanks shorewall-users and

Happy hacking,

James

Fixing jerky scrolling in Firefox

Thu, 11 Apr 2013 05:53:28 +0000

Fedora did a lovely job of updating me to the latest version (v. 20) of Firefox. One problem I found, was that scrolling on certain pages was quite jerky. Performance was worse (or more likely) on pages with a frameset, and pages which were long. Pages with many images made this problem worse.

It turns out that the workaround is to disable hardware acceleration:

After you’ve unchecked this box, restart Firefox, and scrolling is now considerably smoother.

Knowing when to release and deploy your code (...and a mini script)

Mon, 08 Apr 2013 21:27:29 +0000

Knowing when to release and deploy your code can turn into a complicated discussion. In general, In general, I tend to support releasing early and often, for some value of $early and $often. I’ve decided to keep this simple and introduce you to one metric that I use…

I think that I am fairly diligent in adding plenty of comments to my source code. I might even sometimes add too many. I create plenty of XXX, FIXME, or TODO tagged comments as reminders of things to work on.

running your file manager from a terminal

Fri, 22 Mar 2013 23:34:59 +0000

I do a lot of my work in a terminal. For the unfamiliar, this might seem strange, however once you’re comfortable with your shell, this is the best place to be. I don’t restrict myself to it though. I often want to spawn a file manager, or a graphical text editor. When I run nautilus, I usually see something like this:

james@computer:~/some/awesome/directory$ nautilus .
Initializing nautilus-open-terminal extension
Shutting down nautilus-open-terminal extension
james@computer:~/some/awesome/directory$

This is useful, because I can open a file browser right where I want it, it’s annoying, because nautilus runs in that terminal until I close it. (This doesn’t happen if the nautilus process is always running, but since GNOME 3, it isn’t.)

adding range support to python's http server to kickstart with anaconda

Thu, 14 Mar 2013 00:34:19 +0000

I’ve been working on automatic installs using kickstart and puppet. I’m using a modified python httpserver because it’s lightweight, and easy to integrate into my existing python code base. The server was churning away perfectly until anaconda started downloading the full rpm’s for installation. What was going wrong?

Traceback (most recent call last):
[...]
error: [Errno 32] Broken pipe
BorkedError: See TTBOJ for explanation and discussion

As it turns out, anaconda first downloads the headers, and then later requests the full rpm with an http range request. This second range request which begins at byte 1384, causes the “simple” httpserver to bork, because it doesn’t support this more elaborate feature.

learn how to do one minute hacks, in three minutes

Thu, 07 Mar 2013 18:38:45 +0000

I write this technical blog for you to enjoy, and to help me remember. So where do I get all this knowledge? I figure it out! Here’s how I learned to fix a small gedit annoyance in one minute, and within the next three, you’ll be able to do the same for other types of problems too. Ready? Set? Go!

I use gedit enough, that when I hack, I often end up using up more than the five allotted spaces in the “recent files” sections. I wanted to see eight. Since I knew it would have been silly for the developers to hard code the number five, I decided there was a chance that they stored it in the dconf settings. (BTW, there’s also an amazing “Dashboard” plugin which I use for more complex recent-files searching…)

A quick anaconda trick

Mon, 04 Mar 2013 05:00:43 +0000

Here’s a quick anaconda solution that I am now using in some of my kickstart files…

I wanted to bootstrap a machine and do all the partitioning and logical volume creation, but not format or mount one of the logical volumes. The magic parameter I needed was:

--fstype=none

This seems to work perfectly for me. It’s not 100% intuitive to me, but it does work. I hope it’s not an accidental bug in the anaconda code! The full text of my partitioning is:

Automatic hiera lookups in puppet 3.x

Wed, 20 Feb 2013 00:43:35 +0000

Dear readers,

I’ve started the slow migration of code from puppet 2.6 all the way to 3.x+. There were a few things I wasn’t clear on, so hopefully this will help to discuss these and make your migration easier!

I used hiera in 2.6, and I actually like it a lot so far. I was concerned that automatic lookups would pull in values that I wasn’t expecting. This is not the case or a worry. Let’s dive in and let the code speak:

Picking up the pieces after a Fedora 18 install

Tue, 12 Feb 2013 01:15:38 +0000

I love GNOME and Fedora, but “upgrading” from Fedora 17 to 18 did not go well for me. I recommend you wait until either these are all fixed, or Fedora 19+ suits your needs. Here are a list of problems I had, and some workarounds. Hopefully proper patches to these bugs will get merged quickly, so that you don’t need to use these fixes.

Problem: Boot fails after upgrade from Fedora 17 to Fedora 18. I used the new “fedup” method.

How to send and receive files like a professional

Mon, 04 Feb 2013 10:56:35 +0000

Everyone needs to send and receive files sometimes. Traditionally people send files as email attachments. This still works great, and supports encryption, but many mail servers are slow and cap the upper file size limit.

ICQ was a great solution back in the 1990’s, but those days are now over. (I still remember my number.)

A lot of folks use dropbox, which requires a dropbox account, and for you to trust them with your files.

Renaming a GNOME keyring (for seahorse, the passwords and keyrings application)

Sun, 27 Jan 2013 15:33:50 +0000

The GNOME Keyring is a great tool to unify password management across the desktop. Sadly, Firefox is the one application that doesn’t support this natively. (Chrome actually does!)

Seahorse is a useful tool to browse and manage your keyrings. Each keyring is physically stored in: ~/.gnome2/keyrings/$something.keyring

Usually the “$something”, matches the name of the keyring, however the real name comes from within the file. I had an older ubuntu machine running GNOME, and I wanted to import my keyring. Here’s how I did it:

SElinux causes pain when using puppet 2.x with hiera

Thu, 17 Jan 2013 03:03:36 +0000

So hiera wasn’t working when used through my puppetmaster. It worked perfectly when I was running my scripts manually with: puppet apply site.pp but the moment I switched over to regular puppetmasterd usage, everything went dead.

I realized a while back, that I could always expect some hiera failures from time to time. Whether this is hiera’s fault or not is irrelevant, the relevant part is that I quickly added:

Dynamically including classes in puppet

Wed, 16 Jan 2013 07:29:14 +0000

As you might already know, I like pushing the boundaries of what puppet is able to do. Today, I realized that I needed to include a class by variable name. A simple way to do this is possible with:

$foo = 'world'
class { "hello::${foo}::params":
}

I then realized that you could also do this with the standard include keyword:

$foo = 'world'
include "hello::${foo}::params"

If you’re really insane enough to need to have your include depend on a variable, then this should suit your needs. The advantage of the second form is that if that statement appears more than once, you won’t cause a “duplicate definition” error.

Clustering virtual machines with rgmanager and clusvcadm

Thu, 03 Jan 2013 04:25:15 +0000

This could be a post detailing how to host clustered virtual machines with rgmanager and clusvcadm, but that is a longer story and there is much work to do. For now, I will give you a short version including an informative “gotcha”.

With my cluster up and running, I added a virtual machine entry to my cluster.conf:

<vm name="test1" domain="somedomain" path="/shared/vm/" autostart="0" exclusive="0" recovery="restart" use_virsh="1" />

This goes inside the block. As a benchmark, please note that starting the machine with virsh worked perfectly:

How I broke (and fixed) my rgmanager service

Mon, 31 Dec 2012 00:34:45 +0000

Rgmanager, clustat and clusvcadm are useful tools in cluster land. I recently built a custom resource which I added to one of my service chains. Upon inspecting clustat, I noticed:

[root@server1 ~]# clustat
Member Status: Quorate

Member Name                             ID   Status
------ ----                             ---- ------
server1                                 1 Online, Local, rgmanager
server2                                 2 Online, rgmanager

Service Name                   Owner (Last)                   State
------- ----                   ----- ------                   -----
service:service-main-server1   (server1)                      failed

Looking at /var/log/messages, I found:

visible close buttons on every firefox tab (including the last one)

Mon, 26 Nov 2012 05:42:20 +0000

After hacking around with a few firefox internals the other day, I decided there was another little annoyance that I had… When I’m acting insane and using my mouse to open and close tabs, the per tab close button disappears when there is only one tab left! This is apparently a feature, and while I can respect it as a default, it certainly isn’t an option for me. Thankfully, there are a lot of common minded individuals on the net, and the (hack) solution already existed. Here’s to explaining it clearly:

more rows and columns on firefox new tab page

Sat, 24 Nov 2012 19:09:37 +0000

Firefox has a “new tab” speed dial type page available. I use it as my homepage (hint, use: about:newtab) and find it very useful for launching my often used favourites.

My one gripe is that it only shows you a default grid of 3 x 3. You can easily change that if you look in the secret preferences. Open a new tab and type in: about:config, accept the warning, and then search for: browser.newtabpage.rows and browser.newtabpage.columns. These values are easily editable by double clicking on the row. (Bold indicates non defaults.)

hello planet puppet

Fri, 23 Nov 2012 05:59:30 +0000

Hello planet puppet readers!

Brice Figureau[1] who runs planet puppet has just syndicated my blog, so thanks go out to him.

To all readers, I hope you enjoy my content, and please don’t be shy to comment and let me know if there are particular subjects or posts that I should elaborate on. I’ve got a whack of technical posts in the archives, so feel free to browse and let me know what’s good!

recursion in puppet (for no particular reason)

Tue, 20 Nov 2012 06:25:51 +0000

I’m working on some fancy puppet “code”, and I realized recursion could be very useful. I decided to try out a little hack to see if I could get it to work. I’ll jump right into the code:

#!/usr/bin/puppet

define recursion(
    $count
) {
    notify { "count-${count}":
    }
    $minus1 = inline_template('<%= count.to_i - 1 %>')
    if "${minus1}" == '0' {
        notify { 'done counting!':
        }
    } else {
        # recurse
        recursion { "count-${minus1}":
            count => $minus1,
        }
    }
}

# kick it off...
recursion { 'start':
    count => 4,
}

continuous display of log files (better tail -f)

Sun, 18 Nov 2012 01:02:22 +0000

All good sysadmins know about using tail -f to follow a log file. I use this all the time to follow /var/log/messages and my gluster logs in particular. Maybe everyone already knows this, but it deserves a PSA: after a certain amount of time (~days) it seems that new messages don’t appear!

What happens by default is that tail -f follows the file descriptor, not the file name, so when your log files get rotated, the file descriptor still points to the (now renamed) file which no longer gets updates any more.

setting timed events in puppet

Wed, 14 Nov 2012 01:31:43 +0000

I’ve tried to push puppet to its limits, and so far I’ve succeeded. When you hit the kind of bug that forces you to hack around it, you know you are close. In any case, this isn’t about that embarrassing bug, it’s about how to set delayed actions in puppet.

Enter puppet-runonce, a module that I’ve just finished writing. It starts off with the realization that you can exec an action which also writes to a file. If it sees this file, then it knows that it has already completed, and shouldn’t run itself again. The relevant parts are here:

define runonce::exec(
    $command = '/bin/true',
    $notify = undef,
    $repeat_on_failure = true
) {
    include runonce::exec::base

    $date = "/bin/date >> /var/lib/puppet/tmp/runonce/exec/${name}"
    $valid_command = $repeat_on_failure ? {
        false => "${date} && ${command}",
        default => "${command} && ${date}",
    }

    exec { "runonce-exec-${name}":
        command => "${valid_command}",
        creates => "/var/lib/puppet/tmp/runonce/exec/${name}",    # run once
        notify => $notify,
        # TODO: add any other parameters here that users wants such as cwd and environment...
        require => File['/var/lib/puppet/tmp/runonce/exec/'],
    }
}

preventing duplicate parameter values in puppet types

Wed, 07 Nov 2012 18:30:31 +0000

I am writing a keepalived module for puppet. It will naturally be called: “puppet-keepalived”, and I will be releasing the code in the near future! In any case, if you’re familiar with VRRP, you’ll know that each managed link (eg: resource or VIP) has a common routerid and password which are shared among all members in the group. It is important that these parameters are unique across the type definitions on a single node.

changing that keyboard shortcut right there (in gnome)

Sat, 03 Nov 2012 03:12:39 +0000

I love my keyboard shortcuts, and I sometimes I want to change them. If you’re ever in a gnome application and wanted to change that one right there, you can now live-edit them!

In a terminal, first enable this feature:

gsettings set org.gnome.desktop.interface can-change-accels true

Next, hover over the menu item shortcut that you want to change. Enter the shortcut you want. It should update immediately! I like to disable this live-editing, so that I don’t accidentally change any shortcuts. To do so run:

Multifile mode for text editors

Wed, 24 Oct 2012 04:08:38 +0000

Dear internets,

I’m a sysadmin/architect, which means I spend a good amount of time coding in puppet and other languages. Puppet is a great tool, however the puppet community is a bit ~~anal~~ strict about their style policies. I can respect this because I understand how important uniformity is when many developers are sharing code.

(On a side note, I absolutely can’t stand using spaces for indentation, but that’s another story. Please, just tell people to use tabs - preferably with a width of eight spaces.)

Our fusion future

Sat, 20 Oct 2012 03:02:22 +0000

I always find myself reading about fusion. This time, I’ve come across an excellent Q&A on slashdot by some MIT Fusion Researchers. It’s well worth the read, and so is the http://www.fusionfuture.org/ website for some clear explanations about nuclear fusion.

Enjoy,

James

including a recursive tree of files with distutils

Mon, 01 Oct 2012 06:17:58 +0000

It turns out it is non trivial (afaict) to include a tree of files (a directory) in a python distutils data_files argument. Here’s how I managed to do it, while also allowing the programmer to include manual entries:

NAME = 'project_name'
distutils.core.setup(
# ...
    data_files=[
        ('share/%s' % NAME, ['README']),
        ('share/%s' % NAME, ['files/somefile']),
        ('share/%s/templates' % NAME, [
            'files/templates/template1.tmpl',
            'files/templates/template2.tmpl',
        ]),
    ] + [('share/%s/%s' % (NAME, x[0]), map(lambda y: x[0]+'/'+y, x[2])) for x in os.walk('the_directory/')],
# ...
)

finding your software install $prefix from inside python

Thu, 20 Sep 2012 04:12:00 +0000

Good python software developers tend to use distutils and include a setup.py with their code. The problem I often encounter is finding out which prefix your software has been installed in from within the python code. This might be necessary if you want to interact with some data that you’ve installed into: $prefix/share/projectname/ Here are the various steps:

1) Distutils:

NAME='someproject'
distutils.core.setup(
    name=NAME,
    version='0.1',
    author='James Shubin',
    author_email='secret@purpleidea.com',
    url='https://purpleidea.com/',
    description='This is an example project',
    # http://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        'Environment :: Console',
        'Intended Audience :: System Administrators',
        'License :: OSI Approved :: GNU Affero General Public License v3',
        'Operating System :: POSIX :: Linux',
        'Programming Language :: Python',
        'Topic :: Utilities',
    ],
    packages=[NAME],
    package_dir={NAME: 'src'},
    data_files=[
        ('share/%s' % NAME, ['README']),
        ('share/%s' % NAME, ['images/something.png']),
    ],
    scripts=['somebin'],
)

How to avoid cluster race conditions or: How to implement a distributed lock manager in puppet

Thu, 23 Aug 2012 16:14:44 +0000

I’ve been working on a puppet module for gluster. Both this, my puppet-gfs2 module, and other puppet clustering modules all share a common problem: How does one make sure that only certain operations happen on one node at a time?

The inelegant solutions are simple:

Specify manually (in puppet) which node the "master" is, and have it carry out all the special operations. Downside: Single point of failure for your distributed cluster, and you've also written ugly asymmetrical code. Build a beautiful, decentralized setup instead.
Run all your operations on all nodes. Ensure they're idempotent, and that they check the cluster state for success first. Downside: Hope that they don't run simultaneously and race somehow. This is actually how I engineered my first version of puppet-gluster! It was low risk, and I just wanted to get the cluster up; my module was just a tool, not a product.
Use the built-in puppet DLM to coordinate running of these tasks. Downside: Oh wait, puppet can't do this, you misunderstand the puppet architecture. I too thought this was possible for a short duration. Woops! There is no DLM.

Note: I know this is ironic, since by default puppet requires a master node for coordination, however you can have multiple masters, and if they're down, puppetd still runs on the clients, it just doesn't receive new information. (You could also reconfigure your manifests to work around these downsides as they arise, but this takes the point out of puppet: keeping it automatic.)

Mostly elegant: Thoughts of my other cluster crept into my head. Out of nowhere, I realized the solution was: VRRP! You may use a different mechanism if you like, but at the moment I’m using keepalived. Keepalived runs on my gluster pool to provide a VIP for the cluster. This allows my clients to use a highly available IP address to download volume files (mount operation) otherwise, if that particular server were down, they wouldn’t be about to mount. The trick: I tell each node what the expected VIP for the cluster is, and if that IP is present in a facter $ipaddress_, then I let that node execute!

puppet gluster module now in git

Fri, 27 Jul 2012 00:48:50 +0000

The thoughtful bodepd has been kind enough to help me get my puppet-gluster module off the ground and publicized a bit too. My first few commits have been all clean up to get my initial hacking up to snuff with the puppet style guidelines. Sadly, I love indenting my code with tabs, and this is against the puppet rules :(

http://www.emacswiki.org/emacs/TabsSpacesBoth

I’ll be accepting patches by email, but I’d prefer discussion first, especially since I’ve got a few obvious things brewing in my mental queue that should hit master shortly.

a puppet module for gluster

Tue, 24 Jul 2012 15:56:40 +0000

I am an avid cobbler+puppet user. This allows me to rely on my cobbler server and puppet manifests to describe how servers/workstations are setup. I only backup my configs and data, and I regenerate failed machines PRN.

I’ll be publishing my larger cobbler+puppet infrastructure in the future once it’s been cleaned up a bit, but for now I figured I’d post my work-in-progress “puppet-gluster” module, since it seems there’s a real interest.

now syndicated on “planet gluster”

Tue, 24 Jul 2012 15:11:05 +0000

Many thanks to johnmark in #gluster for syndicating my “gluster” tagged blog posts on http://www.gluster.org/blog/

I aim to keep these posts technical and informative, aimed mostly at other sysadmins and gluster users. Please don’t be shy to comment on my writing style or to let me know if you need more information about a particular subject. If you have any ideas about things you’d like me to write about, let me know and I’ll try to do my best. I like feedback!

building intel nic driver (igb) for gluster on centos

Fri, 20 Jul 2012 18:47:50 +0000

I’ve been having some strange networking issues with gluster. “Eco__” from #gluster suggested I try an up to date Intel nic driver. Here are the steps I followed to make that happen. No news yet on if that solved the problem.

Currently my system is using the igb (intel gigabit) driver. To find out what version you are running:

# modinfo -F version igb
3.2.10-k

I found a newer version from the supermicro ftp site. A download and a decompress later, I found an: igb-3.4.7.tar.gz file hiding out. Thanks to the kind people at Intel, this was fairly easy to compile and install. First install some deps:

my gluster setup, described

Thu, 19 Jul 2012 18:26:54 +0000

For the last ~two or so years I’ve played and tested gluster on and off and hanging out in the awesome #gluster channel on Libera.chat. In case you haven’t heard, gluster was acquired by RedHat back in October 2011. This post describes my current setup. I urge you to send your comments and suggestions for improvement. I’ll update this as needed.

Hardware: Ideology: I wanted to build individual self-contained storage hosts. I didn’t want to have servers with separate (serial) attached storage (SAS) like Dell is often pushing. Supermicro fit the design goal, and sold me when I realized I could have the OS drives swappable out the back.

gnome3+others glipper/klipper replacement

Thu, 05 Jul 2012 18:56:40 +0000

So a friend of mine uses kde4 for its klipper feature. Turns out he’s right that it’s an awesome feature! I realized I couldn’t let gnome3 take second place to a clipboard app, so after a bit of searching…

$ sudo yum install gnome-shell-extension-gpaste gpaste xsel

Next hit up: https://extensions.gnome.org/local/ to flip on the extension. I had to first type: ALT-F2, “r” (to restart the gnome shell). Don’t worry your apps won’t die. And then I flipped it on.

fedora 14 to 17 upgrade

Mon, 04 Jun 2012 22:31:41 +0000

I’ve been reluctantly dreading the switch to gnome 3 + shell until it’s been ironed out a little bit more than gnome 3.0 - finally took the plunge. overall it’s working well. here are some (hopefully) useful notes:

preupgrade 14->17 in one step doesn't work. it lets you wait an hour for all the downloads to finish, but once you've rebooted, the preupgrade installer tells you it can't jump this far. fail. reboot into f14, yum clean all, rm -rf /var/cache/yum/preupgrade/* ? and preupgrade to f15. boot into that, and then jumping directly to f17 works great.
remove default f17 ugly fireworks background. the gnome3 default is pretty.
gnome-tweak-tool is essential. try all the options until you figure out which ones you want. listing my personal preferences here is far too boring for the internet.
dash click fix extension was essential for me. also: https://extensions.gnome.org/review/download/1323.shell-extension.zip which does the same thing, but for application launcher entries. this allows gnome-shell to behave properly and launch applications where i want them.
i also added: "Remove user name" and "Remove a11y" extensions.
"system-monitor", is close to what i want, but is not good enough yet. disks access/writes don't show, and it's got a super ugly popup window. old gnome-system-monitor applet was PERFECT. Please bring it back or fix this one.
choose non-ugly background for gdm: # sudo -u gdm dbus-launch gnome-control-center

Hope this helped! Happy hacking,

James

IPMI for linux professionals

Wed, 25 Apr 2012 01:22:22 +0000

The nostalgia of serial console servers, kvm’s and switched PDU’s is hopefully no longer visible in your server room. If not, you must definitely start playing catch up. Please forgive my ignorance, but these things might still be common for big windows shops, however if that’s the case, you’ve got an entirely different set of problems ;)

IPMI is an IP based protocol that allows you to talk directly to a little computer, usually built in to your server. It lets you remotely manage power (on, off, reboot, cycle…) get a serial console, collect sensor readings like temperatures, and do other magical things too if you care to figure them out.

how to use ssh escape characters

Wed, 25 Apr 2012 00:59:01 +0000

So you’ve learned screen, ssh and vim. Time to take your skills to level two.

Day one: You’ve logged in to your server remotely via ssh. You run “screen -xRR”, and two minutes later you’re busy chatting away in irssi and vim is running in the other window, because, you know, real sysadmins don’t use emacs.

Lunch time: You grab your laptop and head off for lunch. When you open the lid and look at your terminals, they’re all frozen, because the tcp connections have died. You force quit the terminals, and you’re back in 30 seconds with new tcp connections.

Dear internets

Wed, 22 Feb 2012 23:04:41 +0000

Dear internets,

During my daily grind of computer architecturing, I often learn something new that I wish a mentor had taught me earlier. I will post my babbles here, so that the interested might learn, and for the wiser to comment on.

Happy Hacking!

James

PS: I had a previous blog, but that server was taken down. It has been merged into this one!

sparse rsync magic

Wed, 22 Feb 2012 18:50:51 +0000

Dear internets,

Once upon a time, I had one of those “brain teaser-of-the-day” calendars. Free business tip: someone should make an ‘rsync’ version. That would be brilliant. (If you don’t know what rsync is, then you should probably go read a different blog.)

Anyone who’s used kvm might know, linux virtual machines can have sparse .raw file backing stores. This means the operating system could think it has one TiB of space, when it is actually sharing a one TiB physical harddrive with many others.

git, gitosis, gitweb and friends...

Thu, 19 Aug 2010 11:03:46 +0000

In case it wasn’t already obvious, I am a huge fan of git, and often prefer it over sliced bread. Recently to help a small team of programmers collaborate, I decided to setup a private git server for them to use. By no claim of mine is the following tutorial unique, however I am writing this to aid those who had trouble following other online tutorials.

Goal: Setup a central git server for private or public source sharing, without having to give everyone a separate shell account.

gnome-do trick

Wed, 31 Mar 2010 12:17:21 +0000

I use gnome-do. It’s fairly useful. One quick trick: press SHIFT - <ENTER> instead of plain <ENTER> to launch without closing the launcher box. Therefore, if you want three terminals, type: <SUPER> - <SPACE>, t(erminal), (<SHIFT> - <ENTER> X 2), <ENTER>.

lecturing and git-bisect

Wed, 31 Mar 2010 10:13:48 +0000

I was recently asked to give a lecture for the PRELUDE series at McGill. Here was my abstract:

I don't like computers, and neither should you.

We spend too much time figuring out how to talk to them, instead of having them figure out how to understand us.

There's a big discontinuity between what software is providing, and the killer features we want!

We're not completely lost though. There are a lot of good tools and methodologies available!

Until the feature gap closes, let me introduce you to some of these tools, and show you how I use the computer.

I spoke about a variety of topics with the intention of filling in everyone's knowledge about the useful tools available to users and developers. I included a section about git-bisect and have posted the script in the examples section of the bash-tutor tarball. It is now available for you to download and share.

I hope everyone enjoyed the lecture, and I always appreciate feedback!

scary cool bash scripting inside a Makefile

Wed, 31 Mar 2010 10:05:21 +0000

Makefiles are both scary and wonderful. When both these adjectives are involved, it often makes for interesting hacking. This is likely the reason I use bash.

In any case, I digress, back to real work. I use Makefiles as a general purpose tool to launch any of a number of shell scripts which I use to maintain my code, and instead of actually having external shell scripts, I just build any necessary bash right into the Makefile.

getting gedit to work like magic

Tue, 09 Mar 2010 17:17:51 +0000

i use gnu/linux. it’s probably no secret. what is more of a secret, is that i secretly (well actually not so secretly) love using gedit for editing text. i still use vim, echo (gnu bash) and emacs (but only for org-mode).

vim is really, really great. but for day to day full-screen coding, i love working in gedit. i only have one [1] longstanding gripe, and today i believe that it is solved. here is the magic combination which appeases my troubled spirit:

n900 features that should be added

Tue, 02 Mar 2010 17:59:47 +0000

i decided i’d compile a short list of features / functionality which should be added to the new nokia N900 to make it totally pro. it’s pretty good as it is, despite it not working with the bell networks. i’ve attempted to order this list in order of importance (more or less).

all the closed source bits must become free software under a GPL or AGPL license.
it should support a fully encrypted file system. if i lose my phone, i shouldn't worry that all my data or contacts get stolen!
addresses that are stored in contacts should integrate with maps program. (currently they only display)
the stock maps tool should integrate nicely with the rest of the phone. currently it's a bit of a sore thumb. confusing too. also it's green.
screen/interfaces should support multi touch.
user should be able to choose username instead of default "user"
hostname should match bluetooth name, etc... on device, one name.
it should support the "usb on the go" host functionality. i'd like to be able to plug in a usb key and be able to copy files to and from it.
better battery, if even only an upgrade to the 1500mAh version.
once plugged in via usb, it would be nice to be able to unmount as usb device while retaining the charging functionality. currently, when you click on the notification area thing, usb mass storage mode is connected, but there is no software way to disconnect. (that i know of) this is important because you can't access the files through your phone when mounted.
user should be able to specify which accounts notify you of new messages and which don't. same for which do periodic updates and when.
default shell should be bash, not busybox, and it should include usual software like "man".
should include stock ir remote control software, that has a large database of hardware including radios, air conditioners and weird tvs. irreco needs some serious work.
people sometimes click on contacts to view data eg a phone number or address to use on a different phone or otherwise. it should be easy to zoom into to read.
my gps should just work. (it doesn't seem to at the moment for some reason)
since i change my wireless WEP keys often, and always forget them, the phones wireless manager should integrate nicely with aircrack-ng to easily help me recover my lost WEP key.
it could be a little thinner. it's a bit of a brick as it stands. i could overlook this problem if it did everything above.
battery life indicator could benefit from having a countdown timer (sure it would change based on usage, but it would give users an approximate idea)
the cameras should be able to function as a usb camera for the computer when the device is plugged in. (over bluetooth could be nice too)

unfortunately i haven't mentioned this phones positive features-- many before me have-- there are many! overall it's a nice package and i look forward to seeing improvements to the above.

a custom epiphany location bar

Mon, 22 Feb 2010 17:04:38 +0000

since i do much research on the internets, i often find myself in a web browser. my favourite of the lot is epiphany. this post isn’t about its merits or failures, but about the awesome way to make my location bar be exactly how i like it: monospaced.

just add the following into your ~/.gtkrc-2.0 file and then restart epiphany. feel free to modify as you wish.

# ~/.gtkrc-2.0
style "Epiphany_Locationbar" {
# leave out the font size if you wish
font_name = "Monospace 12"
#bg[NORMAL] = "#ff0000"
}

# both of these seem to work...
#widget "*.EphyLocationEntry.*" style "Epiphany_Locationbar"
widget_class "*.EphyLocationEntry.*" style "Epiphany_Locationbar"

thanks to raphael for the original post, and the #epiphany developers.

reading obscure documentation

Mon, 08 Feb 2010 13:05:23 +0000

Sometimes the interfaces you need just aren’t well documented, but thanks to the internets, some websearches, and a useful dictionary, you can often decipher random web documentation jargon to help you produce useful code.

getopt vs. optparse vs. argparse

Wed, 03 Feb 2010 12:25:08 +0000

sooner or later you’ll end up needing to do some argument parsing. the foolish end up writing their own yucky parser that ends up having a big if statement filled with things like:

if len(sys.argv) > 1

in it. don’t do this unless you have a really good excuse.

sooner or later, someone directs you to getopt, and you happily continue on with buggy manual parsing thinking you’ve “found the way”. useful in some circumstances, but should generally be avoided.

sorting out the confusion

Wed, 27 Jan 2010 13:03:51 +0000

if i’ve been silent as of late, it’s because i’ve been furiously coding away. i’ve got what i think are some elegant implementations cooking, and with any luck my extra work will pay off in hours and days and months of time saved down the road. i’ve got a few interesting (interesting with respect to your average rating of the blog posts on this site) posts cooking in my mind, and hopefully they’ll appear shortly!

piping data through ssh

Wed, 09 Dec 2009 22:36:05 +0000

not that what i’m about to tell you is brilliant, new or revolutionary, however i thought i’d mention it in case you’re not doing it. also feel free to let me know if there is a better way.

problem: i have some stdout which comes from a command and i want it in a file on another machine.

i could first send it to a temp file, scp that over, and then remove the temp file; but instead, i’ll just:

really well written article

Mon, 16 Nov 2009 13:14:15 +0000

i’m sure that most (if not all) of the technically inclined readers know about this already, but since it’s such a well written article (and detailed in the right places), i felt i should bump up it’s link count.

http://en.wikipedia.org/wiki/Binary_prefix

as an aside, i really liked Knuth’s suggestion; in the meantime let’s use the stupid kibi’s.

IPVS + shorewall

Thu, 05 Nov 2009 11:29:16 +0000

lvs load balancing always felt like an elusive task. here i will document how to get it working with the excellent shorewall firewall, as an extension to their two interface common use case. this was all necessary for a group of grad students that needed to test out and develop some distributed algorithms. it turns out that once you get going, all this is quite easy and fun!

the various components and files used for this setup include:

scary man pages

Mon, 02 Nov 2009 23:21:57 +0000

while doing a little reading and research, i ended up reading a bit of the hdparm man page. never in my life have i read such a scary man page. i guess it is appropriate for halloween, but it came 2 days late in my case. as a result of my being so impressed with the sheer amount of warnings in the manual, i have decided to compile them here.

multi monitors and how it should all work (long babble)

Fri, 23 Oct 2009 21:09:13 +0000

at work i use two screens for my day to day workflow. i find it’s much more efficient for doing work, like coding. i’ll often have at least one terminal open, a full screen text editor, and usually a number of references, such as the dbus specification open in a web browser.

i recently just realized that one of the reasons multiple monitors are so useful to me, isn’t because of the increased screen real estate (although that certaintly is an important factor) but because of how the window manager deals with the windows. for example, when i click maximize, the chosen window expands to fill that monitors capacity, and not the entire set of monitors. it would be nice for this type of behaviour to be configurable somewhere for people with ultra large screens, who want to manage their windows better. the closest i’ve found is: winwrangler which i’ve started using at home where i only have one screen. it works sort of along those lines, however i still want more. multiple “workspaces” are essential. no matter how many screens i have, i’ll always need a few workspaces to be able to flip between email and coding– there is no sense in mixing it all together.

tech support for a keyboard

Fri, 23 Oct 2009 13:14:43 +0000

so my boss was recently kind enough to purchase a new keyboard and mouse combo to attempt to relieve the elbow-and-down pain that i endure due to constant programming and mouse moving. thanks.

i received my “logitech cordless desktop wave pro” today, and 15 seconds later i’m up and running in linux. the keyboard and mouse seem to be really nice, however i won’t dwell on their merits here, i’m sure you can read about that elsewhere on the internets. what i would like to point out is that for some stupid reason, the keyboard only sports a battery indicator light. that’s right, NO NUM LOCK, CAPS LOCKS OR SCROLL LOCK LIGHTS. (woops, sorry didn’t realize caps was on.)

vanilla: my favourite flavour of gnome

Thu, 22 Oct 2009 17:14:47 +0000

after a recent upgrade to ubuntu 9.04, it seems that i’ve been pushed into using some “unfeatures” that i didn’t ask for. well apparently they won’t be unfeatures forever, but for me, notify-osd and indicator-applet aren’t quite ready for my consumption. i realize that ubuntu is trying to improve the desktop experience, but i’m going to need to wait for some more polish first.

so how do i get rid of these horrible things ? i really liked the look of the notification-daemon, and i wanted it back. turns out i can do:

why linux is powerful or: how to erase half your system and then fix it

Wed, 14 Oct 2009 16:00:57 +0000

after a brief bout of stupidity i quickly realized that my makefile had gone awry and was quickly eating through my filesystem.

after ^C killing it, it seems i only took out most of /lib/* and /usr/sbin/* – who needs those anyways… apparently almost everyone.

what happened next. well it turns out i was lucky and had a few shells and a webbrowser open– attempts to launch new programs will fail, but existing programs are already loaded in memory so i was able to work.

the python subprocess module

Wed, 07 Oct 2009 10:24:53 +0000

i’m sure that i won’t be able to tell you anything revolutionary which can’t be found out by reading the manual, but i thought i would clarify it, and by showing you a specific example which i needed.

subprocess.Popen accepts a bunch or args, one of which is the shell argument, which is False by default. If you specify shell=True then the first argument of popen should be a string which is what gets parsed by the shell and then eventually run. (nothing revolutionary)

the power to yield a better console interface

Thu, 20 Aug 2009 13:15:51 +0000

as part of a different project, i needed to duplicate some existing terminal magic in python. what i needed to write was something similar to the getch() function in curses. it can be found in: ncurses*/base/lib_getch.c after doing an: apt-get source libncurses5

what’s the magic? i need to stay in a continuous loop reading from the file descriptor, however i want to return periodically so that gobject doesn’t block and the interface can remain responsive. enter: yield, who comes in and saves the day. see the accompanying code for specifics.

ssh hanging

Wed, 29 Jul 2009 09:40:46 +0000

i’m sure everyone who has used ssh has experienced this, but couldn’t put their finger on what was happening exactly. this article explains.

cheetah == fortran

Tue, 21 Jul 2009 16:59:50 +0000

turns out the cheetah python templating engine (2.0 since year 2006) is quite reminiscent of fortran (since the 1950’s) in their use of the #slurp directive (cheetah) and the $ string. either one, when appended to the end of a string, remove the implicit newline which usually gets printed. it took me ages to figure out how to suppress newline printing back when i did someone’s fortran homework, now i’ve had to struggle with it all over again.

[py]inotify, polling, gtk and gio

Tue, 21 Jul 2009 10:19:52 +0000

i have this software with a gtk mainloop, using dbus and all that fun stuff that seems to play together nicely. i know about the kernel inotify support, and i wanted it to get integrated with that above stack. i thought i was supposed to do it with pyinotify and io_add_watch, but on closer inspection into the pyinotify code it turns out that it seems to actually use polling! (search for: select.poll)

thinking i was confused, i emailed a friend to see if he could confirm my suspicions. we both weren’t 100% sure, a little searching later i was convinced when i found this blog posting. i’m surprised i didn’t find out about this sooner. in any case, my application seems to be happy now.

evanescent 0.3

Tue, 14 Jul 2009 14:53:30 +0000

new evanescent released. get your tar balls today. now with pro dbus support, and happier than 0.2 in every way. please send in any patches and bug reports.

ps: easy one line installation too!

logging out of $SESSION

Mon, 13 Jul 2009 10:05:05 +0000

the software (evanescent) that i’m working on is supposed to log out the user from its current X session. originally i had some yucky looking code that ran a kill on gnome-session, which quickly got replaced with:

os.system('gnome-session-save --logout-dialog')

i’ve decided this was still a little crufty, so i’ve recently replaced this with:

bus = dbus.SessionBus()
remote_object = bus.get_object('org.gnome.SessionManager', '/org/gnome/SessionManager')
# specify the dbus_interface in each call
remote_object.Logout(0, dbus_interface='org.gnome.SessionManager')
# or create an Interface wrapper for the remote object
#iface = dbus.Interface(remote_object, 'org.gnome.SessionManager')
#iface.Logout(0)
# introspection is automatically supported
#print remote_object.Introspect(dbus_interface='org.freedesktop.DBus.Introspectable')

documentation can be found, although it took a little digging. the only catch is that this is gnome specific, and you need different code for kde, and each other DE. thanks to http://www.purinchu.net/wp/2009/06/12/oh-fun/ for the kde version of the above.

convert .ai to .svg using uniconverter

Tue, 16 Jun 2009 11:32:23 +0000

needed to convert a proprietary vector .ai adobe illustrator file to the open .svg format, inkscape failed, ai2svg failed, uniconverted worked! yay. very easy just do:

$ uniconverter Martlet.ai Martlet.svg

brilliant: get rid of mono

Mon, 08 Jun 2009 14:08:14 +0000

fedora announces that they’ll be replacing potentially patent encumbered mono based note taking application tomboy with the line for line port to c++ called gnote.

https://www.redhat.com/archives/fedora-desktop-list/2009-June/msg00003.html

with any luck we'll be able to replace f-spot, and banshee too. have a look at:

http://santanu-sinha.blogspot.com/2009/06/solang.html

coloured tab completion

Tue, 02 Jun 2009 15:06:58 +0000

i <3 tab completion, coloured grep, and coloured ls, however i would love to see coloured tab completion.

what would this entail? imagine you have files: “apply”, “applied” and “applications” in a directory, and you typed: “cat ap” <TAB>, then i would expect to see: apply applied applications (highlighting the rest of the words / options. or perhaps there could be a mode that just highlights the next letter that needs to be typed.

grep, shopt:dotglob and (hidden) dot files

Tue, 02 Jun 2009 11:56:08 +0000

i thought grep was broken, but it’s not. see below:

james@dazzle:~$ echo $0
bash
james@dazzle:~$ echo 'hello world' > .hello
james@dazzle:~$ grep 'hello world' *
james@dazzle:~$ grep 'hello world' .*
.hello:hello world
james@dazzle:~$ shopt | grep dotglob
dotglob            off
james@dazzle:~$ shopt -s dotglob
james@dazzle:~$ shopt | grep dotglob
dotglob            on
james@dazzle:~$ grep 'hello world' *
.hello:hello world
james@dazzle:~$ shopt -u dotglob
james@dazzle:~$ shopt | grep dotglob
dotglob            off
james@dazzle:~$ rm .hello
james@dazzle:~$

the problem as it turns out is that the glob character `*’ (the asterisk) doesn’t expand to include dot files unless you have the shopt variable set. so you can either use to workaround shown above or set it. personally i’ll keep mine off.

Server Fault

Wed, 27 May 2009 11:10:37 +0000

Server Fault, the question and answer forum for systems administrators opened their doors today to the public. It is created by the same notorious bloggers who brought you Stack Overflow, Joel on Software and the Daily WTF.

Go have a look, ask a question, answer a question and have fun!

community

Wed, 27 May 2009 09:36:11 +0000

so the idea was to open this up to sys admins in other mcgill departments, so if we don’t know about you, drop us a line and we’ll get you added to the blog. if people like the idea, then we’ll host this on a better cs machine than dazzle.

first babble

Fri, 22 May 2009 16:16:04 +0000

hello! this blog is intended to be a place where ramblings and discoveries, by and for systems administrators and programmers can be posted and seen by many to ultimately help in the dissemination of useful findings that we wish we had been able to have read here earlier.

if that didn’t make too much sense, perhaps i could rephrase it.

Dinner

Mon, 01 Jan 0001 00:00:00 +0000

This page has some more information about the annual dinner.

Statistics

Year	Meals served	Location
2025	921 (565+356)	Concordia
2024	980 (550+430)	Concordia
2023	893 (584+309)	St. Ignatius of Loyola
2022	> 731 (counting stopped before the end)	St. Ignatius of Loyola
2021	cancelled	N/A
2020	0 (covid)	N/A
2019	1126	St. Ignatius of Loyola
2018	1097 (742+355)	St. Ignatius of Loyola
2017	1035	St. Ignatius of Loyola
2016	899	St. Ignatius of Loyola
2015	1222	St. Ignatius of Loyola
2014	~850?	St. Ignatius of Loyola
2013	872 (shannon)	St. Ignatius of Loyola
2012	852 (shannon)	St. Ignatius of Loyola
2011	762 (shannon)	St. Ignatius of Loyola
?	?	Chez Mes Amis
1994	?	St. Columba
1988	125?	?

History

In 1988, a group of individuals got together to run an annual free dinner for anyone in need. The dinner welcomed poor, homeless, lonely, low income, or anyone otherwise in need of a free meal.

Dinner

Mon, 01 Jan 0001 00:00:00 +0000

Every year on December 25th, a free meal is served to the community. We welcome poor, homeless, lonely, low income, or anyone otherwise in need of a free meal.

The date coincides with christmas day, but the event is non-religious, and is open to anyone of any faith, including agnostics and atheists.

The event is run entirely by volunteers using donated time, money and food. We typically serve a meal consisting of turkey, mashed potatoes, stuffing, cooked vegetables, cranberry sauce, gravy, and dessert. All of the items except for the turkey are 100% vegetarian. (We have our standard turkey gravy and a vegetarian alternative.) We’ll typically have additional donated dishes in limited quantities, many of which have included vegan options as well.)

DJ

Mon, 01 Jan 0001 00:00:00 +0000

I DJ occasionally. Mostly for fun, as I like to put on a good show. I invest almost 100% of my performance fees into paying for show consumables (fire, smoke, bubbles, etc…) and for buying new equipment to improve the shows.

I play almost entirely Old School Hip-Hop and Rap, combined with some Break Beats and occasional slow jams.

If you’d like to hire me to play at your bar, club, or venue, please contact me. I’m happy to play weddings, funerals, and other private events as well. I’m happy to perform at fundraising events for non-profits at cost if I can.

Donate

Mon, 01 Jan 0001 00:00:00 +0000

If you like my blog, and you like my code, please leave me a tip!

You can also donate money via my patreon.

Funds and other resources give me more time to spend on code and articles for you!

I’m often in need of hardware, so if you have something you can donate or sponsor, please let me know.

If your company wants to fly me somewhere to come hack on my code in your office, that could be arranged. Maybe you need help installing it, integrating it, or adding a feature?

How to cook a turkey

Mon, 01 Jan 0001 00:00:00 +0000

What follows are instructions for cooking a turkey. This page is mostly intended for folks that donate turkeys to the annual dinner.

“Let’s Talk Turkey”

(based on an article from Gourmet Magazine, November 2005)

Instructions

If frozen, make sure the turkey has been thawing the refrigerator for one or two days depending on the size of the turkey. You can remove it one hour before cooking to allow it to warm up even more.
Pre-heat oven to 230 Celsius (450F).
Remove feathers and quills (if any).
Some turkeys will contain a bag with the neck and organs (giblets) inside the turkey. Remove them and cook outside the turkey.
Rinse turkey inside and out and pat dry.
Mix salt and pepper in a small bowl and sprinkle evenly in turkey cavities and over the skin.
Put turkey on oven rack in lower third of oven.
Rotate turkey 180 degrees (π radians) half-way through roasting.
Use an instant-read thermometer inserted into the fleshy part of the thigh close to (but not touching) the bone.
Also test the temperature in the turkey’s breast.
Check that the space where the drumstick touches the body is roasted. Sometimes, because it’s hidden, it doesn’t roast as fast as the rest of the bird.
Temperature must register 77 Celsius (170F) for turkey to be cooked.
Once out of oven let turkey stand for 30 minutes. Temperature of thigh meat should rise to 82 Celsius (180F).
They really do come out tender and juicy.

Notes

Don’t stuff the turkey. If you do, then by the time the stuffing is at a safe temperature to eat, the turkey will be well overcooked.
Don’t cover it with tented aluminum foil.
All ovens are different but an 8 kg (16 pound) turkey should be done in about 75 to 90 minutes. But always check the temperature.
Please make sure to not add bacon, milk, butter, soy, or any other meats or ingredients other than salt, pepper and vegetables. We try to make sure folks who can’t eat those items can enjoy the turkey as well.
Please do not carve the turkey. Our group of volunteers will do so on site to make sure that all the donated turkeys are carved similarly. (So that they all get reheated at the same rate.) You can either bring it in fresh from the oven, or you can refrigerate it and bring it in cold. Please do not freeze it after cooking as we won’t be able to use it.
Make sure you use a meat thermometer to ensure the turkey is cooked in the centre to a minimum of 77 degrees celsius. (170 degrees fahrenheit.)
Save the turkey drippings, they help make good gravy.

Les Garderies K.I.D.S. Infrastructure

Mon, 01 Jan 0001 00:00:00 +0000

Les Garderies K.I.D.S. is a chain of daycares in the Montreal region. I completed a number of projects for this group. Below you can find a select list of projects, some associated photos, and testimonials.

Video Viewing Screens

One of the video viewing screens.

Daycare staff needed access to live video streams of different areas of the daycares so that they can keep watch of everything going on. For this I installed a number of PoE cameras alongside custom video viewing screens which were placed around the daycares.

Mentoring Program

Mon, 01 Jan 0001 00:00:00 +0000

I run a very small mentoring program. The format has been evolving over time. The program is intended to be very difficult, but fulfilling. Some information about the program can be found on this page.

I currently offer two streams, each with a different specialization:

Stream A: Software Engineering & Programming
Stream B: Systems Administration & DevOps

I started this program because I was disappointed with the quality of many new hackers these days, and in particular the shift away from strong basic skills. I wanted to give something back in return for everyone who helped point me in the right direction over the years.

Mentoring Program

Mon, 01 Jan 0001 00:00:00 +0000

Some time ago, I started a small mentoring program, to help interested learners improve their computer skills. I’ve taken on a small number of participants to keep quality high, and I think I’ve had some great results. Here are some of their testimonials.

Testimonials

When I decided to get back into IT after more than half a decade away from the field, I was in dire need of some direction and James came to the rescue. Within a week I was becoming proficient in Go and getting experience designing elegant solutions to real world problems. James was an extremely effective mentor, technically and professionally. At the end of the program I secured a fantastic job at a consulting firm, and have been very happy there for nearly two years.

– Jonathan

I have been participating in the mentorship program for about a year. I joined the mentorship program to improve my knowledge of systems programming and open source development. Purpleidea (My mentor) has been very helpful in facilitating my learning. The program uses a hands-off approach that allows for the necessary struggling to develop problem solving and self-learning skills. I would recommend this to anyone interested in open source development, Golang or systems programming.

– Donald

James introduced me to the world of GNU+Linux and has guided me through learning its ins and outs. Ever since, I truly can't imagine going back to proprietary operating systems. Through hacking with him, he has helped me learn the best practices of free / open source development, which is an invaluable skill that can be hard to learn all alone.

– Adam

James is a great mentor since on top of being knowledgable he is also able to effectively understand the skills that you can improve on and guide you towards the goal of becoming a better hacker based on your current skill level. He also has the ability to know when to give hints and when not to give hints which enriched my learning experience under him. I was happy to have him as a mentor since I could see my skills in Golang and Git increase significantly. He is also a fun person to hack with so the more you can meet with him the better.

– Raunak

More Information

More information about the program is available here.

Mgmt Config

Mon, 01 Jan 0001 00:00:00 +0000

After having spent well over five years using and writing Puppet, I decided that I had some designs for a next generation automation tool. This eventually became what is now known as Mgmt Config or mgmt.

Mgmt runs as a distributed system.

The project has a number of novel features, including an event-based execution engine, a reactive domain specific language (DSL) and the ability to execute in parallel. A cluster of mgmt agents can self-coordinate and run as a distributed system using the raft consensus algorithm.

Open Technical Problems

Mon, 01 Jan 0001 00:00:00 +0000

What follows is my personal list of what I’ve decided to call “Open Technical Problems”. Each of these items is a description of an open technical problem that I think would be really excellent to have solved. I believe all of these could be reasonably easy to solve if someone has the time, money, and will to do so.

If you are working on any of these ideas, of if one is solved, please let me know!

Pendeli's Pizza Cote-St-Luc

Mon, 01 Jan 0001 00:00:00 +0000

Their telephone operator usually sounds very unhappy, but their pizza is fairly good. Ask them not to burn it! I recommend the “cheese and tomato”, aka plain. Tell them The Technical Blog of James sent you.

Here’s the most recent menu:

I posted this because there’s no menu to be found online.

Happy Hacking!

Seymour Smith - Dorothy

Mon, 01 Jan 0001 00:00:00 +0000

My favourite piece of piano music is called “Dorothy” by “Seymour Smith”. My grandmother [1] used to play it to me when I was young. When she died, I asked if I could keep the piano music.

If you know anything about Seymour Smith, have a copy of any of his sheet music, or would like to send me a recording of Dorothy of any of his other music, please contact me.

The Technical CAPTCHA

Mon, 01 Jan 0001 00:00:00 +0000

This was intended to be a CAPTCHA for recruiters.

The CAPTCHA

Comment or write a question on one of my technical articles.
Contact me.
If your comment was good, you’ll get a reply back!

It need not be long, but you do get points for cleverness if it asks an interesting question, or makes light of a fun corner case that I haven’t covered. The clever part about this, is that you get to think of it as a way of quizzing me, and I get to engage with people whose interest is technology.

Wisdom

Mon, 01 Jan 0001 00:00:00 +0000

An ideal language allows us to express easily what is useful for the programming task, and at the same time makes it difficult to write what leads to incomprehensible or incorrect programs.

– Nico Habermann

yesiscan

Mon, 01 Jan 0001 00:00:00 +0000

yesiscan is a tool for performing automated license scanning. It takes many forms of user input, then runs a bunch of different analysis tools in parallel, and finally produces a customizable report which can easily be shared with your team. It has been designed to make license analysis simple for users, and easy to manage and configure for centralized legal teams.

yesiscan has a friendly web ui.

It can be used via a CLI, web ui, http API, or even imported as a library and built into your existing software.