The Technical Blog of James


This is the new home of The Technical Blog of James!

Formerly available here, all posts have been migrated to the archives.


Leaving Red Hat Posted on Apr 4, 2018
purpleidea is moving on...


I’ve spent about four years at Red Hat, and now it’s time to move on… TL;DR: had to leave Red Hat and start Patreon to fund mgmt. What follows is a bit of historical rambling, and some forward looking statements. Retrospective: Long-time readers of my blog will know that I was very active in the puppet ecosystem for many years. I learned a lot while writing puppet code, and while building some of my outrageous puppet hacks.

Read More

Running `make` from anywhere Posted on Mar 10, 2018
Run commands from your `Makefile`, even if you're nested deeply!


Sometimes while I’m deep inside mgmt’s project directory, I want to run an operation from the Makefile which lives in the root! Unfortunately, if you do so while nested, you’ll just get: [email protected]:~/code/mgmt/resources$ make build make: *** No rule to make target 'build'. Stop. The Ten Minute Solution: I figured I’d hack out a quick solution. What I came up with looks like this: #!/bin/bash # James Shubin, 2018 # run `make` in the first directory (or its parent recursively) that it works in MF='Makefile' # looks for this file, could look for others, but that's silly CWD=$(pwd) # starting here while true; do if [ -e "$MF" ]; then make [email protected] # run make!

Read More

Mgmt Configuration Language Posted on Feb 5, 2018
Introducing the Mgmt Configuration Language


Mgmt has given us a lot of great features, demos, and insight into how infrastructure automation should be managed. Unfortunately it wasn’t a complete tool yet, because it was missing a crucial piece for its completeness. That all changes today with the addition of: the language. An ideal language allows us to express easily what is useful for the programming task, and at the same time makes it difficult to write what leads to incomprehensible or incorrect programs.

Read More

Hello World! Posted on Nov 13, 2017
First test of new blog


Hello! This is a new version of my website and blog. If you experience any issues, please let me know! Happy Hacking, James

Read More

Copyleft is Dead. Long live Copyleft! Posted on Oct 17, 2017


As you may have noticed, we recently re-licensed mgmt from the AGPL (Affero General Public License) to the regular GPL. This is a post explaining the decision and which hopefully includes some insights at the intersection of technology and legal issues. Disclaimer: I am not a lawyer, and these are not necessarily the opinions of my employer. I think I’m knowledgeable in this area, but I’m happy to be corrected in the comments.

Read More

Extracting movies from libreoffice Posted on Jul 12, 2017


I have a short movie that I imported into a libreoffice presentation. I wanted a copy of that movie back, but I couldn’t figure out how to extract a copy. In desperation, I figured I’d try opening the file with file-roller, the GNOME archive manager. [email protected]:/tmp$ file mgmt-berlin-osdc-17may2017.odp mgmt-berlin-osdc-17may2017.odp: OpenDocument Presentation [email protected]:/tmp$ mkdir out [email protected]:/tmp$ file-roller -f mgmt-berlin-osdc-17may2017.odp -e out/ [snip] [email protected]:/tmp$ cd out/ [email protected]:/tmp/out$ ls Configurations2/ Media/ meta.xml Pictures/ styles.

Read More

Declarative vs. Imperative paradigms Posted on May 5, 2017


Recently, while operating two different remote-controlled appliances, I realized that it was high time for a discussion about declarative and imperative paradigms. Let’s start by looking at the two remotes: Two different “remotes”. The one on the left operates a television, and the one on the right controls a central heating and cooling system.At first glance you will notice that one of these remotes is dark, and the other is light.

Read More

Osmocom femtocell un-boxing Posted on Apr 30, 2017


LaForge and the fine folks at Osmocom (Sysmocom) recently had a femtocell giveaway. I didn’t expect to have much time to hack on things, but they were still quite generous in sending me one. It arrived, and I took some un-boxing photos for anyone who is curious. A box arrived in the mail…Which recurses into an inner box…Inner box is box like.Finally… The unit is displayed.Here it is in all its glory.

Read More

Metaparameters in mgmt Posted on Mar 1, 2017


In mgmt we have meta parameters. They are similar in concept to what you might be familiar with from other tools, except that they are more clearly defined (in a single struct) and vastly more powerful. In mgmt, a meta parameter is a parameter which is codified entirely in the engine, and which can be used by any resource. In contrast with Puppet, require/before are considered meta parameters, whereas in mgmt, the equivalent is a graph edge, which is not a meta parameter.

Read More

Faster golang builds Posted on Feb 26, 2017


I’ve been hacking in golang since before version 1.4, and the speed at which my builds finished has been mostly trending downwards. Let’s look into the reasons and some fixes. TL;DR click-bait title: “Get 4x faster golang builds with this one trick!”. Here are the three reasons my builds got slower: The compiler Before version 1.5, the compiler was written in C but with that release, it moved to being pure golang.

Read More

Ten minute hacks: Process pause & resume Posted on Jan 6, 2017


I’m old school and still rocking an old X220 laptop because I didn’t like the new ones. My battery life isn’t as great as I’d like it to be, but it gets worse when some “webapp” (which I’d much rather have as a native GTK+ app) causes Firefox to rev my CPU with their websocket (hi gmail!) poller. This seems to happen most often on planes or when I’m disconnected from the internet.

Read More

Send/Recv in mgmt Posted on Dec 7, 2016


I previously published “A revisionist history of configuration management”. I meant for that to be the intro to this article, but it ended up being long enough that it deserved a separate post. I will explain Send/Recv in this article, but first a few clarifications to the aforementioned article. Clarifications I mentioned that my “revisionist history” was inaccurate, but I failed to mention that it was also not exhaustive! Many things were left out either because they were proprietary, niche, not well-known, of obscure design or simply for brevity.

Read More

A revisionist history of configuration management Posted on Nov 30, 2016


I’ve got a brand new core feature in mgmt called send/recv which I plan to show you shortly, but first I’d like to start with some background. History This is my historical perspective and interpretation about the last twenty years in configuration management. It’s likely inaccurate and slightly revisionist, but it should be correct enough to tell the design story that I want to share. Sometime after people started to realize that writing bash scripts wasn’t a safe, scalable, or reusable way to automate systems, CFEngine burst onto the scene with the first real solution to this problem.

Read More

Remote execution in mgmt Posted on Oct 7, 2016


Bootstrapping a cluster from your laptop, or managing machines without needing to first setup a separate config management infrastructure are both very reasonable and fundamental asks. I was particularly inspired by Ansible’s agent-less remote execution model, but never wanted to build a centralized orchestrator. I soon realized that I could have my ice cream and eat it too. Prior knowledge If you haven’t read the earlier articles about mgmt, then I recommend you start with those, and then come back here.

Read More

mgmt has a logo Posted on Sep 30, 2016


The mgmt config project got a logo! The full commit is here. Thanks to Sarah Jane Cox for creating it. Happy Hacking, James PS: I might have a few stickers to give out too! Ask me next time you see me if you’d like one! Alternatively, use the artwork to make your own and share with your friends!

Read More

Live dmesg following Posted on Aug 29, 2016


All good sysadmins eventually learn about using tail -F to tail files. Yes upper-case F is superior. Around the time I wrote that article, I remember wanting to stream dmesg output too! The functionality wasn’t available without some sort of polling hack, but it turns out that kernel support for this actually landed around the same time in version 3.5.0! Most GNU/Linux distros are probably running a new enough version by now, and you can now dmesg –follow (or dmesg -w):

Read More

Seen in downtown Montreal... Posted on Aug 3, 2016


The Technical Blog of James was seen on an outdoor electronic display in downtown Montreal! Thanks to one of my readers for sending this in. I guess the smart phone revolution is over, and people are taking to reading my articles on bigger screens! The “poutine” is decent proof that this is probably Montreal.If you’ve got access to a large electronic display, put up the blog, snap a photo, and send it my way!

Read More

Ten minute hacks: Hacking airplane headphones Posted on Jul 18, 2016


I was stuck on a 14 hour flight last week, and to my disappointment, only one of the two headphone speakers were working. The plane’s media centre has an audio connector that looks like this: Someone should consider probing this USB port.The hole to the left is smaller than a 3.5mm headphone jack, and designed for a proprietary headphone connector that I didn’t have, and the two holes to the right are part of a different proprietary connector which match with the cheap airline headphones to provide the left and right audio channels.

Read More

Automatic clustering in mgmt Posted on Jun 20, 2016


In mgmt, deploying and managing your clustered config management infrastructure needs to be as automatic as the infrastructure you’re using mgmt to manage. With mgmt, instead of a centralized data store, we function as a distributed system, built on top of etcd and the raft protocol. In this article, I’ll cover how this feature works. Foreword: Mgmt is a next generation configuration management project. If you haven’t heard of it yet, or you don’t remember why we use a distributed database, start by reading the previous articles:

Read More

Upcoming speaking In Hong Kong and South Africa Posted on Jun 20, 2016


I’m thrilled to tell you that I’ll be speaking about mgmt in Hong Kong and South Africa. It will be my first time to both countries and my first time to Asia and Africa! In Hong Kong I’ll be speaking at HKOSCon2016. In South Africa I’ll be speaking at DebConf16. I’m looking forward to meeting with many of the hard-working Debian hackers, and collaborating with them to build and promote excellent Free Software.

Read More

One hour hacks: Remote LUKS over SSH Posted on Apr 25, 2016


I have a GNU/Linux server which I mount a few LUKS encrypted drives on. I only ever interact with the server over SSH, and I never want to keep the LUKS credentials on the remote server. I don’t have anything especially sensitive on the drives, but I think it’s a good security practice to encrypt it all, if only to add noise into the system and for solidarity with those who harbour much more sensitive data.

Read More

Automatic grouping in mgmt Posted on Mar 30, 2016


In this post, I’ll tell you about the recently released “automatic grouping” or “AutoGroup” feature in mgmt, a next generation configuration management prototype. If you aren’t already familiar with mgmt, I’d recommend you start by reading the introductory post, and the second post. There’s also an introductory video. Resources in a graph Most configuration management systems use something called a directed acyclic graph, or DAG. This is a fancy way of saying that it is a bunch of circles (vertices) which are connected with arrows (edges).

Read More

Automatic edges in mgmt Posted on Mar 14, 2016


It’s been two months since I announced mgmt, and now it’s time to continue the story by telling you more about the design of what’s now in git master. Before I get into those details, let me quickly recap what’s happened since then. Mgmt community recap: I gave the first public presentation about mgmt at CfgMgmtCamp. I repeated the talk at DevConf.cz. The video recording is available. Felix wrote about his work cross compiling puppet code to mgmt.

Read More

Introducing: git tpush Posted on Feb 16, 2016


On today’s issue of “one hour hacks”, I’ll show you how you can stop your git drive-by’s to git master from breaking your CI tests… Let’s continue! The problem: Sometimes I’ve got a shitty one-line patch that I want to push to git master. I’m usually right, and everything tests out fine, but usually isn’t always, and then I look silly while I frantically try to fix git master on a project that I maintain.

Read More

Debugging golang programs Posted on Feb 15, 2016


I’ve been writing a lot of golang lately. I’ve hit painful problems in the past. Here are some debugging tips. Hopefully they help you out. I bet you don’t know #2. #0 Use log.Printf: This should go without saying, but I’m ashamed to say it’s what I use the most. We’ve only been C programming for 44+ years, and it’s still what is most useful! #1 Use go run -race:

Read More

Next generation configuration mgmt Posted on Jan 18, 2016


It’s no secret to the readers of this blog that I’ve been active in the configuration management space for some time. I owe most of my knowledge to what I’ve learned while working with Puppet and from other hackers working in and around various other communities. I’ve published, a number, of articles, in an, attempt, to push, the field, forwards, and to, share the, knowledge, that I’ve, learned, with others. I’ve spent many nights thinking about these problems, but it is not without some chagrin that I realized that the current state-of-the-art in configuration management cannot easily (or elegantly) solve all the problems for which I wish to write solutions.

Read More

Upcoming speaking Posted on Jan 18, 2016


I’ve got a few upcoming speaking engagements. If you’ll be attending one of these events, come see me or any of the other excellent speakers! Please remember to check the official schedules in case there are any changes! I’ll be speaking at the Brussels CentOS Dojo: Automated Infrastructure Testing with Oh-My-Vagrant …and the CentOS CI Time/date unconfirmed: I’ll be showing some CI tricks, and showing you how the CentOS CI is the perfect CI for multi-machine test environments.

Read More

Trying out Ceph with Oh-My-Vagrant Posted on Dec 28, 2015


Daniel P. Berrangé wrote about trying out a single node ceph cluster. I decided to take his article and turn it into an Oh-My-Vagrant omv.yaml file. It took me about two minutes to do so, and two hours to debug a problem caused by something I had broken on my laptop. If you’d like to replicate his article in less than 5 minutes, pull down the omv.yaml file that I’ve just published and run omv up.

Read More

Matching arbitrary URL's to custom Firefox profiles Posted on Nov 14, 2015


We’re constantly clicking on all sorts of different URL’s throughout the day. These clickable links appear in webpages (including in “web apps” like gmail) in mail clients like Evolution, in terminals such as GNOME-terminal, and any other GTK+ app on your GNU/Linux desktop. I wanted to perform custom actions when arbitrary URL’s are clicked, including running certain links in separate Firefox profiles. There are a bunch of different steps you have to do to get this working, but it should be easy to follow along.

Read More

Thanking Oh-My-Vagrant contributors for version 1.0.0 Posted on Aug 18, 2015


The Oh-My-Vagrant project became public about one year ago and at the time it was more of a fancy template than a robust project, but 188 commits (and counting) later, it has gotten surprisingly useful and mature. [email protected]:~/code/oh-my-vagrant$ git rev-list HEAD --count 188 [email protected]:~/code/oh-my-vagrant$ git log $(git log --pretty=format:%H|tail -1) commit 4faa6c89cce01c62130ef5a6d5fa0fff833da371 Author: James Shubin <[email protected]> Date: Thu Aug 28 01:08:03 2014 -0400 Initial commit of vagrant-puppet-docker-template... This is an attempt to prototype a default environment for vagrant+puppet+docker hacking.

Read More

Vagrant and Oh-My-Vagrant on RHEL7 Posted on Aug 11, 2015


My employer keeps paying me, which I appreciate, so it’s good to spend some time to make sure RHEL7 customers get a great developer experience! So here’s how to make vagrant, vagrant-libvirt and Oh-My-Vagrant work on RHEL 7+. The same steps should work for CentOS 7+. I’ll first paste the commands you need to run, and then I’ll explain what’s happening for those that are interested: # run these commands, and then get hacking!

Read More

Making an empty RPM Posted on Aug 11, 2015


I am definitely not an RPM expert, in fact, I’m afraid of it, but with recent tools such as COPR, and my glorious Makefile, some aspects of it have become palatable. This post will be about a recent journey I had building the most useless RPM ever. A video of my journey building this RPM.Because of reasons, I wanted to satisfy an RPM dependency for a package that I wanted to install without rebuilding that RPM.

Read More

Golang parallelism issues causing “too many open files” error Posted on Jul 27, 2015


I’ve been hacking in golang for a while, but I’ll admit that I didn’t get too deep into some of the language nuances until more recently. Since some of them have started to bite me, here’s a little post-mortem of one of the problems I was having. After hacking and testing code all day, I made a seemingly innocuous change, and when running my program, I saw the following error:

Read More

Git archive with submodules and tar magic Posted on Jul 23, 2015


Git submodules are actually a very beautiful thing. You might prefer the word powerful or elegant, but that’s not the point. The downside is that they are sometimes misused, so as always, use with care. I’ve used them in projects like puppet-gluster, oh-my-vagrant, and others. If you’re not familiar with them, do a bit of reading and come back later, I’ll wait. I recently did some work packaging Oh-My-Vagrant as RPM’s.

Read More

Oh-My-Vagrant “Mainstream” mode and COPR RPM's Posted on Jul 8, 2015


Making Oh-My-Vagrant (OMV) more developer accessible and easy to install (from a distribution package like RPM) has always been a goal, but was previously never a priority. This is all sorted out now. In this article, I’ll explain how “mainstream” mode works, and how the RPM work was done. (I promise this will be somewhat interesting!) Prerequisites: If you haven’t read any of the previous articles about Oh-My-Vagrant, I’d recommend you start there.

Read More

A super privileged Puppet container Posted on Jun 12, 2015


In this new crazy world of containers and immutable hosts, one might still want to run previous generation software such as Puppet on a current generation Atomic host. This article will explain how you can do that, and offer some proof of concept code. The atomic host doesn’t provide a yum or dnf command, because the software is pre-baked into a read-only /usr/ partition. To “install” (to use) additional software, it usually needs to be distributed and run as a container.

Read More

Kubernetes clusters with Oh-My-Vagrant Posted on May 2, 2015


I’ve added the ability to deploy a Kubernetes cluster with Oh-My-Vagrant (omv). I’ve also built an automated developer experience so that you can test your Kubernetes powered app in minutes. If you want to redeploy a new version, or see how your app behaves during a rolling update, you can use omv to test this out in minutes! I’ve recorded a screencast (~15 min), if you’d like to see some of this in action.

Read More

Docker containers in Oh-My-Vagrant Posted on Apr 20, 2015


The Oh-My-Vagrant (omv) project is an easy way to bootstrap a development environment. It is particularly useful for spinning up an arbitrary number of virtual machines in Vagrant without writing ruby code. For multi-machine container development, omv can be used to help this happen more naturally. Oh-My-Vagrant can be very useful as a docker application development environment. I’ve made a quick (<9min) screencast demoing this topic. Please have a look:

Read More

Sharing dev environments with Oh-My-Vagrant Posted on Apr 8, 2015


With Oh-My-Vagrant (omv) you can set up a dev environment in seconds. (Read the omv introduction if you’ve never used it before!) Since everything is defined in a single omv.yaml file, it is easy to share your cluster prototype with a friend! The one missing feature was associating code with this config file. This is now possible! Let me show you how it works… In the omv.yaml file there is an extern variable.

Read More

Fancy git aliases and git cherryfetch Posted on Mar 16, 2015


Here are two quick git tricks that I’ve added to my toolbox lately… I wanted to create a git alias that takes in argv from the command, but in the middle of the command. Here’s the hack that I came up with for the [alias] section of my ~/.gitconfig: [alias] # cherryfetch fetches a repo ($1) / branch ($2) and applies it rebased! # the && true at the end eats up the appended args cherryfetch = !

Read More

Building RHEL Vagrant Boxes with Vagrant-Builder Posted on Feb 23, 2015


Vagrant is a great tool for development, but Red Hat Enterprise Linux (RHEL) customers have typically been left out, because it has been impossible to get RHEL boxes! It would be extremely elegant if hackers could quickly test and prototype their code on the same OS as they’re running in production. Secondly, when hacking on projects that have a long initial setup phase (eg: a long rpm install) it would be excellent if hackers could roll their own modified base boxes, so that certain common operations could be re-factored out into the base image.

Read More

Introducing: Silent counter Posted on Feb 9, 2015


You might want to write code that can tell how many iterations have passed since some action occurred. Alternatively, you might want to know if it’s the first time a machine has run Puppet. To do these types of things, you might wish to have a monotonically increasing counter in your Puppet manifest. Since one did not exist, I set out to build one! The code: If you just want to try the code, and skip the ramble, you can include common::counter into your manifest.

Read More

Replying to mailing lists with Evolution Posted on Nov 27, 2014


I use the Evolution mail client. It does have a few annoying bugs, but it has a plethora of great features too! Hopefully this post will inspire you to help hack on this piece of software and fix the bugs! Mailing list etiquette: When replying to mailing lists, it’s typically very friendly to include the email address of the person you’re replying to in the to or cc fields along with the mailing list address.

Read More

Captive web portals are considered harmful Posted on Nov 27, 2014


Recently, when I tried to access http://slashdot.org/ in Firefox, I would see my browser title bar flash briefly to “AT&T GUI”, and then I would get redirected to: http://slashdot.org/cgi-bin/redirect.ha which returns slashdot’s custom error 404 page! What is going on? (Read on for answer…) Did slashdot mess up their mod_rewrite config? (Nope, works fine in a different browser…) Did my HTTPS everywhere extension go crazy? (Nope, still broken when disabled…) Are my HTTP requests being MITM-ed?

Read More

The switch as an ordinary GNU/Linux server Posted on Nov 4, 2014


The fact that we manage the switches in our data centres differently than any other server is patently absurd, but we do so because we want to harness the power of a tiny bit of silicon which happens to be able to dramatically speed up the switching bandwidth. beware of proprietary silicon, it’s absurd!That tiny bit of silicon is known as an ASIC, or an application specific integrated circuit, and one particularly well performing ASIC (which is present in many commercially available switches) is called the Trident.

Read More

Testing Evolution's git master and GNOME continuous Posted on Oct 22, 2014


I’ve wanted a feature in Evolution for a while. It was formally requested in 2002, and it just recently got fixed in git master. I only started publicly groaning about this missing feature in 2013, and mcrha finally patched it. I tested the feature and found a small bug, mcrha patched that too, and I finally re-tested it. Now I’m blogging about this process so that you can get involved too!

Read More

Hacking out an Openshift app Posted on Oct 18, 2014


I had an itch to scratch, and I wanted to get a bit more familiar with Openshift. I had used it in the past, but it was time to have another go. The app and the code are now available. Feel free to check out: https://pdfdoc-purpleidea.rhcloud.com/ This is a simple app that takes the URL of a markdown file on GitHub, and outputs a pandoc converted PDF. I wanted to use pandoc specifically, because it produces PDF’s that were beautifully created with LaTeX.

Read More

Continuous integration for Puppet modules Posted on Oct 10, 2014


I just patched puppet-gluster and puppet-ipa to bring their infrastructure up to date with the current state of affairs… What’s new? Better README’s Rake syntax checking (fewer oopsies) CI (testing) with travis on git push (automatic testing for everyone) Use of .pmtignore to ignore files from puppet module packages (finally) Pushing modules to the forge with blacksmith (sweet!) This last point deserves another mention. Puppetlabs created the “forge” to try to provide some sort of added value to their stewardship.

Read More

Fixing dropbox “conflicted copy” problems Posted on Sep 30, 2014


I usually avoid proprietary cloud services because of freedom, privacy and vendor lock-in concerns. In addition, there are some excellent libre (and hosted) services such as WordPress, Wikipedia and OpenShift which don’t have the above problems. Thirdly, there are every day Free Software tools such as Fedora GNU/Linux, Libreoffice, and git-annex-assistant which make my computing much more powerful. Finally, there are some hosted services that I use that don’t lock me in because I use them as push-only mirrors, and I only interact with them using Free Software tools.

Read More

Introducing: Oh My Vagrant! Posted on Sep 3, 2014


If you’re a reader of my code or of this blog, it’s no secret that I hack on a lot of puppet and vagrant. Recently I’ve fooled around with a bit of docker, too. I realized that the vagrant, environments I built for puppet-gluster and puppet-ipa needed to be generalized, and they needed new features too. Therefore… Introducing: Oh My Vagrant! Oh My Vagrant is an attempt to provide an easy to use development environment so that you can be up and hacking quickly, and focusing on the real devops problems.

Read More

Rough data density calculations Posted on Aug 27, 2014


Seagate has just publicly announced 8TB HDD’s in a 3.5” form factor. I decided to do some rough calculations to understand the density a bit better… Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since I always work in base 2, but I hate the -bi naming conventions. Seagate is most likely announcing an 8TB HDD, which is actually smaller than a true 8TiB drive.

Read More

Hybrid management of FreeIPA types with Puppet Posted on Jul 24, 2014


(Note: this hybrid management technique is being demonstrated in the puppet-ipa module for FreeIPA, but the idea could be used for other modules and scenarios too. See below for some use cases…) The error message that puppet hackers are probably most familiar is: Error: Duplicate declaration: Thing[/foo/bar] is already declared in file /tmp/baz.pp:2; cannot redeclare at /tmp/baz.pp:4 on node computer.example.com Typically this means that there is either a bug in your code, or someone has defined something more than once.

Read More

One minute hacks: the nautilus scripts folder Posted on Jun 26, 2014


Master SDN hacker Flavio sent me some tunes. They were sitting on my desktop in a folder: $ ls ~/Desktop/ uncopyrighted_tunes_from_flavio/ I wanted to listen them while hacking, but what was the easiest way…? I wanted to use the nautilus file browser to select which folder to play, and the totem music/video player to do the playing. Drop a file named totem into: ~/.local/share/nautilus/scripts/ with the contents: #!/bin/bash # o hai from purpleidea exec totem -- "[email protected]" and make it executable with:

Read More

Securely managing secrets for FreeIPA with Puppet Posted on Jun 6, 2014


Configuration management is an essential part of securing your infrastructure because it can make sure that it is set up correctly. It is essential that configuration management only enhance security, and not weaken it. Unfortunately, the status-quo of secret management in puppet is pretty poor. In the worst (and most common) case, plain text passwords are found in manifests. If the module author tried harder, sometimes these password strings are pre-hashed (and sometimes salted) and fed directly into the consumer.

Read More

Hiera data in modules and OS independent puppet Posted on Jun 4, 2014


Earlier this year, R.I.Pienaar released his brilliant data in modules hack, a few months ago, I got the chance to start implementing it in Puppet-Gluster, and today I have found the time to blog about it. What is it? R.I.’s hack lets you store hiera data inside a puppet module. This can have many uses including letting you throw out the nested mess that is commonly params.pp, and replace it with something file based that is elegant and hierarchical.

Read More

Restarting GNOME shell via SSH Posted on May 29, 2014


When GNOME shell breaks, you get to keep both pieces. The nice thing about shell failures in GNOME 3, is that if you’re able to do a restart, the active windows are mostly not disturbed. The common way to do this is to type ALT-F2, r, <ENTER>. Unfortunately, you can’t always type that in if your shell is very borked. If you are lucky enough to have SSH access, and another machine, you can log in remotely and run this script:

Read More

Vagrant on Fedora with libvirt (reprise) Posted on May 13, 2014


Vagrant has become the de facto tool for devops. Faster iterations, clean environments, and less overhead. This isn’t an article about why you should use Vagrant. This is an article about how to get up and running with Vagrant on Fedora with libvirt easily! Background: This article is an update of my original Vagrant on Fedora with libvirt article. There is still lots of good information in that article, but this one should be easier to follow and uses updated versions of Vagrant and vagrant-libvirt.

Read More

Keeping git submodules in sync with your branches Posted on May 6, 2014


This is a quick trick for making working with git submodules more magic. One day you might find that using git submodules is needed for your project. It’s probably not necessary for everyday hacking, but if you’re glue-ing things together, it can be quite useful. Puppet-Gluster uses this technique to easily include all the dependencies needed for a Puppet-Gluster+Vagrant automatic deployment. If you’re a good hacker, you develop things in separate feature branches.

Read More

Working at RedHat Posted on Apr 2, 2014


So this happened: James just James at RedHat headquarters in North Carolina wearing his new red hat.RedHat made me an offer, and I am happy to say that I have just started this week! I am proud to have joined a company that employs many of the worlds foremost, highly professional and clever hackers. It is indubitably the best Free Software [1] / Open Source company out there, and they ship some of the greatest and most elegant software available.

Read More

Puppet-Gluster now available as RPM Posted on Mar 27, 2014


I’ve been afraid of RPM and package maintaining [1] for years, but thanks to Kaleb Keithley, I have finally made some RPM’s that weren’t generated from a high level tool. Now that I have the boilerplate done, it’s a relatively painless process! In case you don’t know kkeithley, he is a wizard [2] who happens to also be especially cool and hardworking. If you meet him, be sure to buy him a $BEVERAGE.

Read More

Introducing Puppet Exec['again'] Posted on Mar 24, 2014


Puppet is missing a number of much-needed features. That’s the bad news. The good news is that I’ve been able to write some of these as modules that don’t need to change the Puppet core! This is an article about one of these features. Posit: It’s not possible to apply all of your Puppet manifests in a single run. I believe that this holds true for the current implementation of Puppet.

Read More

Preserving your working directory in gnome-terminal Posted on Mar 20, 2014


I use gnome-terminal for most of my hacking. In fact, I use it so much, that I’ll often have multiple tabs open for a particular project. Here’s my workflow: Control+Alt+t (My shortcut to open a new gnome-terminal window.) cd ~/code/some_cool_hack/ # directory of some cool hack Control-Shift-t (Shortcut to open a new gnome-terminal tab.) Hack, hack, hack… The problem is that the new tab that I’ve created will have a $PWD of ~, instead of keeping the $PWD of ~/code/some_cool_hack/, which is the project I’m working on!

Read More

Speaking at SCALE today! Posted on Feb 21, 2014


I’ll be giving a talk at SCALE today about automatically deploying GlusterFS with Puppet-Gluster and Vagrant. I’ll be giving some live demos, and this will cover some of the material from: Automatically deploying GlusterFS with Puppet-Gluster + Vagrant! and it will contain excerpts from: Screencasts of Puppet-Gluster + Vagrant I’ll also be talking about some new upcoming features, and am happy to answer all of your questions! The talk will be part of Infrastructure.

Read More

Building a snow shelter Posted on Feb 17, 2014


To give you a break from the usual GNU/Linux/DevOps/Puppet/GlusterFS drab, I’ve decided to have a go at writing a different kind of technical article. This article will show you how to build the traditional Canadian snow dwelling known as a quinzee. If you will be travelling to Canada, I recommended that you read through this article ahead of time, so that you don’t offend your host by being unfamiliar with their traditional living accommodations.

Read More

Scathing review of the Lenovo X240 Posted on Feb 2, 2014


I’m using a Lenovo X201 with 8GiB of RAM. Apart from some minor issues, I’ve been very satisfied with this laptop. It’s over four years old, and so I decided to see what’s available on the horizon. I did not buy an X240 because of the following reasons: The X240 has only one slot for RAM and thus supports a maximum of 8GiB. I think it’s pretty ridiculous for any successor to the X230 to support less RAM.

Read More

Show the exit status in your $PS1 Posted on Jan 29, 2014


As an update to my earlier article, a friend gave me an idea of how to make my $PS1 even better… First, the relevant part of my ~/.bashrc: ps1_prompt() { local ps1_exit=$? if [ $ps1_exit -eq 0 ]; then #ps1_status=`echo -e "\[\033[32m\]"'\$'"\[\033[0m\]"` ps1_status='\$' else ps1_status=`echo -e "\[\033[1;31m\]"'\$'"\[\033[0m\]"` fi ps1_git='' if [ "$(__git_ps1 %s)" != '' -a "$(__git_ps1 %s)" != 'master' ]; then ps1_git=" (\[\033[32m\]"$(__git_ps1 "%s")"\[\033[0m\])" fi PS1="${debian_chroot:+($debian_chroot)}\[email protected]\h:\[\033[01;34m\]\w\[\033[00m\]${ps1_git}${ps1_status}" } # preserve earlier PROMPT_COMMAND entries.

Read More

Screencasts of Puppet-Gluster + Vagrant Posted on Jan 27, 2014


I decided to record some screencasts to show how easy it is to deploy GlusterFS using Puppet-Gluster+Vagrant. You can follow along even if you don’t know anything about Puppet or Vagrant. The hardest part of this process was producing the actual videos! If recommend first reading my earlier articles if you’re planning on following along: Vagrant on Fedora with libvirt Vagrant vsftp and other tricks Vagrant clustered SSH and ‘screen’ Automatically deploying GlusterFS with Puppet-Gluster + Vagrant!

Read More

Building base images for Vagrant with a Makefile Posted on Jan 20, 2014


I needed a base image “box” for my Puppet-Gluster+Vagrant work. It would have been great if good boxes already existed, and even better if it were easy to build my own. As it turns out, I wasn’t able to satisfy either of these conditions, so I’ve had to build one myself! I’ve published all of my code, so that you can use these techniques and tools too! Status quo: Having an NIH problem is bad for your vision, and it’s best to benefit from existing tools before creating your own.

Read More

Testing GlusterFS during “Glusterfest” Posted on Jan 16, 2014


The GlusterFS community is having a “test day”. Puppet-Gluster+Vagrant is a great tool to help with this, and it has now been patched to support alpha, beta, qa, and rc releases! Because it was built so well (*cough*, shameless plug), it only took one patch. Okay, first make sure that your Puppet-Gluster+Vagrant setup is working properly. I have only tested this on Fedora 20. Please read: Automatically deploying GlusterFS with Puppet-Gluster+Vagrant!

Read More

Automatically deploying GlusterFS with Puppet-Gluster + Vagrant! Posted on Jan 8, 2014


Puppet-Gluster was always about automating the deployment of GlusterFS. Getting your own Puppet server and the associated infrastructure running was never included “out of the box”. Today, it is! (This is big news!) I’ve used Vagrant to automatically build these GlusterFS clusters. I’ve tested this with Fedora 20, and vagrant-libvirt. This won’t work with Fedora 19 because of bz#876541. I recommend first reading my earlier articles for Vagrant and Fedora: Vagrant on Fedora with libvirt Vagrant vsftp and other tricks Vagrant clustered SSH and ‘screen’ Once you’re comfortable with the material in the above articles, we can continue…

Read More

Vagrant clustered SSH and ‘screen’ Posted on Jan 2, 2014


Some fun updates for vagrant hackers… I wanted to use the venerable clustered SSH (cssh) and screen with vagrant. I decided to expand on my vsftp script. First read: Vagrant on Fedora with libvirt and Vagrant vsftp and other tricks to get up to speed on the background information. Vagrant screen: First, a simple screen hack… I often use my vssh alias to quickly ssh into a machine, but I don’t want to have to waste time with sudo-ing to root and then running screen each time.

Read More

Vagrant vsftp and other tricks Posted on Dec 21, 2013


As I previously wrote, I’ve been busy with Vagrant on Fedora with libvirt, and have even been submitting, patches and issues! (This “closed” issue needs solving!) Here are some of the tricks that I’ve used while hacking away. Default provider: I should have mentioned this in my earlier article but I forgot: If you’re always using the same provider, you might want to set it as the default. In my case I’m using vagrant-libvirt.

Read More

Vagrant on Fedora with libvirt Posted on Dec 9, 2013


Apparently lots of people are using Vagrant these days, so I figured I’d try it out. I wanted to get it working on Fedora, and without Virtualbox. This is an intro article on Vagrant, and what I’ve done. I did this on Fedora 19. Feel free to suggest improvements. Intro: Vagrant is a tool that easily provisions virtual machines, and then kicks off a configuration management deployment like Puppet. It’s often used for development.

Read More

Now syndicated on “Planet Fedora” Posted on Dec 9, 2013


I’m now syndicated on the Fedora Project planet. If you haven’t read through my blog yet, let me introduce myself, I’m James, and I write The Technical Blog of James. I’m a sysadmin, DevOps/Puppet hacker, I.T./network architect and physiologist. Hi! I run Fedora as my primary desktop, but I also use it for servers, particularly for development before future versions of RHEL and CentOS release. I’m most well-known for Puppet-Gluster, but I’ve also written a decent Puppet-IPA (FreeIPA) module.

Read More

Advanced recursion and memoization in Puppet Posted on Nov 27, 2013


As a follow-up to my original article on recursion in Puppet, and in my attempt to Push Puppet (to its limit), I’ll now attempt some more advanced recursion techniques in Puppet. In my original recursion example, the type does recurse, but the callee cannot return any value to the caller because it is a type, and not strictly a function. This limitation immediately limits the usefulness of this technique, but I’ll try to press on!

Read More

Documentation for Puppet-Gluster Posted on Nov 20, 2013


Ironically, one of the reasons that I started writing Puppet code, was so that I could spend more time designing and building, and less time writing documentation. I suppose I’m a victim of my success, because Puppet-Gluster has grown large enough to warrant its own documentation. So I gave in, and put together some documentation. It’s available as markdown, and, also as a pdf. As before, there is an examples/ directory which you might want to use as additional reference.

Read More

Iteration in Puppet Posted on Nov 17, 2013


People often ask how to do iteration in Puppet. Most Puppet users have a background in imperative programming, and are already very familiar with for loops. Puppet is sometimes confusing at first, because it is actually (or technically, contains) a declarative, domain-specific language. In general, DSL’s aren’t always Turing complete, nor do they need to support loops, but this doesn’t mean you can’t iterate. Until recently, Puppet didn’t have an explicit looping construct, and it is quite possible to build complex modules without using this new construct.

Read More

Pushing Puppet at Puppet Camp DC, LISA 2013 Posted on Nov 5, 2013


Hi there, I hope you enjoyed my “Pushing Puppet (to its limit)” talk and demos from Puppet Camp D.C., LISA 2013. As requested, I’ve posted the code and slides. Here is the code: https://github.com/purpleidea/puppet-pushing This module will require three modules as dependencies. The dependencies are: My Puppet-Common module https://github.com/purpleidea/puppet-common My Puppet-Runonce module https://github.com/purpleidea/puppet-runonce My Puppet-FSM module https://github.com/purpleidea/puppet-fsm Each example doesn’t require all the dependencies, so if you’re only interested in the FSM, you only need that module.

Read More

Gluster Community Day, LISA 2013, Monday Posted on Nov 5, 2013


I’m here at LISA 2013 at the Gluster Community Day. I’ve been asked by Joe Brockmeier to give a little recap about what’s been going on. So here it is! Wesley Duffee-Braun started off with a nice overview talk about GlusterFS. The great thing about his talk was that he gave a live demo, running on virtual machines, on his laptop. If you’re a new GlusterFS user, this is good exposure to help you get started.

Read More

Speaking at LISA 2013 about Puppet and GlusterFS Posted on Nov 3, 2013


I’m speaking at LISA 2013, the “Large Installation System Administration” conference. This conference runs all week in Washington. I’ll be giving two talks during the week, and attending at least one BOF. My first talk is on Monday during the Gluster Community Day. I’ll be speaking about puppet-gluster, and giving a live demo. I’ll be showing some new features too. If you’d like to talk more about puppet-gluster, or want to attend the talk, give me a shout, or sign up at the above Gluster Community Day link.

Read More

Easier strace of scripts with pidof -x Posted on Oct 26, 2013


Here’s a one minute read, about a trick which I discovered today: When running an strace, it’s common to do something like: strace -p<pid> Smarter hackers know that they can use some bash magic and do: strace -p`pidof <process name>` However, if you’re tracing a script named foo.py, this won’t work because the real process is the script’s interpreter, and pidof python, might return other unrelated python scripts.

Read More

first release of puppet-shorewall Posted on Oct 21, 2013


Oh, hi there. In case you’re interested, I’ve just made a first release of my puppet-shorewall module. This isn’t meant as an exhaustive shorewall module, but it does provide most of the usual functionality that most users need. In particular, it’s the module dependency that I use for many of my other puppet modules that provide firewalling. This is probably where you’re most likely to consume it. In general most modules just implement shorewall::rule, so if you really don’t want to use this code, you can implement that signature yourself, or not use automatic firewalling.

Read More

Desktop Notifications for Irssi in Screen through SSH in Gnome Terminal Posted on Oct 18, 2013


I’m usually on IRC, but I don’t often notice incoming pings until after the fact. I had to both write, and modify various scripts to get what I wanted, but now it’s all done, and you can benefit from my hacking by following along… The Setup Laptop -> Gnome-Terminal -> SSH -> Screen -> Irssi This way, I’m connected to IRC, even when my laptop isn’t. I run irssi in a screen session on an SSH server that I manage, and I use gnome-terminal on my laptop.

Read More

GNOME Montreal Summit Posted on Oct 15, 2013


This October 12th to 14th Montreal hosted the GNOME boston summit. Many thanks to Canonical for sponsoring breakfast, Savoir Faire Linux for hosting a great 6 à 10 with fancy snacks, and RedHat for sponsoring a pool night. What follows is some technical commentary about stuff that went on. JHBuild JHBuild is a tool to make it easy to download/clone (from git) and compile all the GNOME modules and applications. It was easy to get going.

Read More

Show current git branch in PS1 when branch is not master Posted on Oct 10, 2013


Short post, long command… I’ve decided to start showing the current git branch in my PS1. However, since I don’t want to know when I’m on master, I had to write a new PS1 that I haven’t yet seen anywhere. Add the following to your .bashrc: PS1='${debian_chroot:+($debian_chroot)}\[email protected]\h:\w\$ ' if [ -e /usr/share/git-core/contrib/completion/git-prompt.sh ]; then . /usr/share/git-core/contrib/completion/git-prompt.sh PS1='${debian_chroot:+($debian_chroot)}\[email protected]\h:\w$([ "$(__git_ps1 %s)" != "" -a "$(__git_ps1 %s)" != "master" ] && (echo -e " (\[33[32m\]"$(__git_ps1 "%s")"\[33[0m\])") || echo "")\$ ' fi

Read More

Finite state machines in puppet Posted on Sep 28, 2013


In my attempt to push puppet to its limits, (for no particular reason), to develop more powerful puppet modules, tobuild in a distributed lock manager, and to be more dynamic, I’m now attempting to build a Finite State Machine (FSM) in puppet. Is this a real finite state machine, and why would you do this? Computer science professionals might not approve of the purity level, but they will hopefully appreciate the hack value.

Read More

No Ads! Posted on Sep 26, 2013


I just shelled out to wordpress.com to buy the No Ads upgrade. I think they’re good people. I hope you all appreciate it. Let me know if you do. Happy hacking, James

Read More

Installing missing GNOME games Posted on Sep 26, 2013


I just realized that my Fedora 19 installation didn’t have any of the GNOME games installed by default any more. I guess there’s no love for nibbles. Here’s a quick one-liner to get them all back: $ sudo yum search game | grep gnome | awk '{print $1}' | xargs sudo yum install -y Loaded plugins: etckeeper, langpacks, refresh-packagekit Package gnome-nibbles-3.8.0-2.fc19.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package gnome-chess.

Read More

Bittorent sync for repository mirroring Posted on Sep 23, 2013


Theron Conrey writes about using: BitTorrent Sync as Geo-Replication for Storage We got a chance to talk about this idea at Linuxcon. I’m not entirely convinced there aren’t some problem edge cases with this solution, but I think it will be hard to tell as long as the BitTorrent sync library is proprietary. I did come up with a special case of Theron’s idea that I believe could work well.

Read More

Gluster Community Day, Thursday Posted on Sep 20, 2013


I’m here in New Orleans hacking up a storm and getting to meet fellow gluster users IRL. John Mark Walker started off with a great “State of the GlusterFS union” style talk. Today Louis (semiosis) gave a great talk about running glusterfs on amazon. It was highly pragmatic and he explained how he chose the number of bricks per host. The talk will be posted online shortly. Marco Ceppi from Canonical gave a talk about juju and gluster.

Read More

Linuxcon day three, Wednesday Posted on Sep 18, 2013


After hacking away on Monday and Tuesday and meeting fellow nerds IRL, I’ve landed even more changes to puppet-gluster. My git master branch now sits at 47 commits. $ git clone https://github.com/purpleidea/puppet-gluster.git Cloning into 'puppet-gluster'... remote: Counting objects: 317, done. remote: Compressing objects: 100% (144/144), done. remote: Total 317 (delta 187), reused 275 (delta 148) Receiving objects: 100% (317/317), 82.17 KiB | 12.00 KiB/s, done. Resolving deltas: 100% (187/187), done. $ cd puppet-gluster/ $ git log | grep '^commit' | wc -l 47 $ git log | head commit <a href="https://github.

Read More

Linuxcon day two, Tuesday Posted on Sep 18, 2013


Continuing on from yesterday, I’ve met even more interesting people. I chatted with Dianne Mueller about some interesting ideas for gluster+openshift. More to come on that front soon. Hung out with Jono Bacon and talked a bit about puppet-gluster on Ubuntu. If there is interest in the community for this, please let me know. Thanks to John Mark Walker and RedHat for sponsoring me and introducing me to many of these folks.

Read More

Linuxcon day one, Monday Posted on Sep 18, 2013


I’m here in New Orleans at Linux Con, hacking on puppet-gluster and talking to lots of interesting folks. I’ve met gluster hacker Theron Conrey, and my host John Mark Walker, Fedora and Raspberry Pi experts Spot and Ruth Suehle, and many others too. The hotel is very nice. The bathroom sink has two taps of course, but both of them are hot. The New Orleans heat is probably the cause of this.

Read More

New puppet-gluster features before Linuxcon Posted on Sep 8, 2013


Hey there, I’ve done a bit of puppet-gluster hacking lately to try to squeeze some extra features and testing in before Linuxcon. Here’s a short list: SELinux fixes to keep Dan Walsh happy :) Ping and status checks before volume creation. Now puppet-gluster will be less noisy about failures or missing executions that are due to the necessary incremental nature of puppet-gluster runs. You’ll need multiple puppet runs to get a complete setup, so don’t let puppet complain part way through.

Read More

Puppet-Gluster and me at Linuxcon Posted on Sep 2, 2013


John Mark Walker, (from Redhat) has been kind enough to invite me to speak at the Linuxcon Gluster Workshop in New Orleans. I’ll be speaking about puppet-gluster, giving demos, and hopefully showing off some new features. I’m also looking forward to meeting up with gluster expert Joe Julian. If there are features that puppet-gluster is missing, or you have a use case that I haven’t covered, please let me know, and I’ll try to work on it for you ahead of the conference.

Read More

Finding YAML errors in puppet Posted on Aug 25, 2013


I love tabs, they’re so much easier to work with, but YAML doesn’t like them. I’m constantly adding them in accidentally, and puppet’s error message is a bit cryptic: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: malformed format string - %S at /etc/puppet/manifests/foo.pp:18 on node bar.example.com This happens during a puppet run, which in my case loads up YAML files. The tricky part was that the error wasn’t at all related to the foo.

Read More

Upgrading from Fedora 18 to Fedora 19 Posted on Aug 4, 2013


It was time to take the plunge and upgrade from Fedora 18 to Fedora 19. Fedora 18 was one of the worst releases ever, so I figured it could only get better. I ran my backups as usual, however this time I didn’t seem to need them, the upgrade process went off without a hitch! I used the fedup-cli process over the network. I always run these things inside of screen.

Read More

a puppet-ipa user type and a new difference engine Posted on Jul 9, 2013


A simple hack to add a user type to my puppet-ipa module turned out to cause quite a stir. I’ve just pushed these changes out for your testing: 3 files changed, 1401 insertions(+), 215 deletions(-) You should now have a highly capable user type, along with some quick examples. I’ve also done a rewrite of the difference engine, so that it is cleaner and more robust. It now uses function decorators and individual function comparators to help wrangle the data into easily comparable forms.

Read More

Fresh releases! puppet-ipa, puppet-nfs, puppet-gluster Posted on Jun 23, 2013


I’ve been a little slow in making release announcements, so here’s some news: I’ve just released the third stage of my puppet-ipa module. At the moment it now supports installation, managing of hosts, and managing of services. It integrates with my puppet-nfs module to allow you to easily setup and run an NFSv4 kerberized server and client. While we’re at it, that’s some more news: I’ve just released a puppet-nfs module to make your /etc/exports management easier.

Read More

Playing with FreeIPA and puppet Posted on Jun 18, 2013


So I just rolled a new vm to hack around with FreeIPA. Here are some things that I’ve come across so far. I was planning on configuring LDAP, and Kerberos manually, but the included webui looks like a lovely tool to have for the data entry, user administrator type who likes to click on things. Let’s explore… /etc/hosts: FreeIPA is choosy about how your /etc/hosts is formatted. It requires an entry that has a particular order, that is:

Read More

Oh hello...

Hey, hacker!
You should follow me to get new updates!