The Technical Blog of James


This is the new home of The Technical Blog of James!

Formerly available here, all posts have been migrated to the archives.


Bash $PATH filtering Posted on Mar 2, 2024
How to filter things out of bash completion


As most modern GNU+Linux distro users already know, you get a lot of tools included for free! Many of these may clutter up your $PATH and make bash tab completion more difficult than it has to be. Here’s a way to improve this! A mess: Here’s what I see when I tab-complete cd<TAB>: james@computer:~$ cd cd cddb_query cd-info cd-read cd-convert cd-drive cd-it8 cdrecord cd-create-profile cd-fix-profile cdmkdir cdtmpmkdir cdda-player cd-iccdump cd-paranoia I genuinely only use three of those commands.

Read More

Upcoming Speaking and Events Posted on Jan 31, 2024
Where to get your mgmt fix in the next month


There are three upcoming mgmtconfig events happening within the next seven days! mgmt @ FOSDEM: On February 3rd, I’ll be giving a talk in the golang devroom at FOSDEM. Please come early (1hr+) if you want to get a seat, usually this room is especially full. This happens in Brussels, but it should be recorded if there aren’t any technical glitches. (I got rejected for the main track, but it’s my fault because I need to make my work more generally appealing first.

Read More

Git branch filtering Posted on Jan 28, 2024
How to make git branch filtering more palatable


If you have more than one git branch up in the air at the same time, having a system to track them and filter them is essential. Here are the stages I went through with git branch. xkcd reminds us that we're insanePrefixes: My first tip involves using a memorable prefix. For new features I name my branches with feat/. For example feat/my-new-feature. Yes the slash and the dashes are treated like any regular character.

Read More

Five minute hacks: Swapping left and right headphone audio in wireplumber Posted on Nov 22, 2023
How to actually make this happen.


I’m on Fedora 37 at the moment. My headphone audio is backwards. How do we fix it? I’ve apparently never found the right magic until now. The Setup: I have a crappy “couch computer”. It’s used for casual internet browsing and video watching. It has an analog audio out which goes to my speakers. I’d like to use headphones for when I don’t want to annoy my neighbours. I have a pair of wireless headphones that I got for free, but the battery doesn’t last longer than an hour.

Read More

One minute hacks: Saving time inserting images in Libreoffice Posted on Aug 2, 2023
How to anchor as character by default.


You’ve probably used libreoffice. You opened a document, and proceeded to insert an image into the text. It wiggles all over and never rests where you want it to. The solution: You most likely want the “anchor as character” option. Right-click on the newly inserted image, go to the Anchor menu item, and choose As Character. You will have to do this for each image you insert which is annoying…

Read More

UEFI booting, and with a new kernel module Posted on Jul 24, 2023
How to get your dracut-built initramfs machine to boot with UEFI when you're missing a kernel module.


Picture this. You’ve got a supermicro server. It’s got eight hot-swappable 3.5” HDD’s. You want to use all of those for storage, so you buy a nifty add-on that let’s you add four additional 2.5” SSD’s. Those are great for your OS. That’s a great way to keep your data separate. The motherboard only supports eight drives, so you’ll need an HBA for the additional four… What are some catches and things you need to watch out for?

Read More

Improving git diff output in 5 min Posted on Jun 1, 2023
How to get rid of the annoying a/ and b/ prefixes.


Everyone uses git diff. But there’s that odd annoyance that you’ve probably seen before but are so used to that you’ve forgotten about it. The a/ and b/ prefixes… Example: james@computer:~/code/mgmt (feat/cool-feature)$ git diff diff --git a/lang/funcs/core/os/readfile_func.go b/lang/funcs/core/os/readfile_func.go index 206ba798d..48ac29dc7 100644 --- a/lang/funcs/core/os/readfile_func.go +++ b/lang/funcs/core/os/readfile_func.go @@ -228,6 +228,8 @@ func (obj *ReadFileFunc) Stream() error { // Close runs some shutdown code for this function and turns off the stream. func (obj *ReadFileFunc) Close() error { + // TODO: Port to the new ctx-in-Stream close API.

Read More

Blocked select logging (deadlock finding) in golang Posted on May 26, 2023
How to log when you're waiting for a blocked select statement...


I’ve been writing some massively concurrent code for mgmt lately. The concurrency makes it incredibly bug prone. In this case it’s usually races or deadlocks. I’ve got a new trick to make the process more pleasant. The Problem: I’m sure many of you have written some code like this before: // just a small excerpt for { select { case <-someEvent: // some awesome thing happened! case <-ctx.Done(): return ctx.

Read More

Deadline context test cancellation in golang Posted on Feb 24, 2023
How to map the cancellation context into your golang tests...


I decided to write a fancy test harness in golang today. The test wraps a big internal engine for mgmt and at the top-level it takes a context for cancellation. If you don’t know about the context package, then you should go understand that and then come back here… Don’t feel bad, I had no idea what it was about at first either! The Problem: I assumed there would be some way to follow a notification from the test runner down into my test to tell it when it was time to cleanup and exit early… I expected that making my own ^C signal handler wouldn’t be correct, and I (incorrectly) assumed that the interface I’d be looking for would offer a golang context that I could pass into my code.

Read More

Magic typing with shortcuts on Linux Posted on Jun 16, 2021
How to workaround bad software by automating your typing.


Ever find yourself typing more characters than is necessary when using some annoying application or website? Here’s a quick hack around that. Intro: There’s a magic program called xdotool written by Jordan Sissel that solves this problem. I’ve known about his excellent work for years, but only today did I learn he also wrote xdotool. It’s named as such because it tells the X server to do something for you.

Read More

USB Audio on a Boombox Posted on Jan 11, 2021
Getting USB audio working on a Sony ZS-RS60BT


In an earlier post, I got USB audio working on my used, manual, 2013 Honda Civic EX. Today I’ll do the same for a Sony ZS-RS60BT boombox. Here it is in all its used glory after a little cleaning.Background: I decided I should get a small portable boombox to practice my dope dance moves. I currently still suck because I think I spend more time computering than out on the floor.

Read More

Working at Amazon Posted on Dec 28, 2020
I found a new job


TL;DR: I’ve got a new job working for Amazon, in the “Open Source Program Office”. I’m now an Amazoner! More information below… The Amazon. Source: https://en.wikipedia.org/wiki/Amazon_rainforestBackground: A little over three months ago, I decided to get back into full-time work. For the past 2.5 years since I left my job at Red Hat, I’ve been mostly working full-time on Mgmt Config and my DevOps/SWE Mentoring Program. To do such things, I had to live off of my savings, which isn’t sustainable long-term.

Read More

James is available for hire! Posted on Sep 10, 2020
Looking For A Job


TL;DR: I’m available for hire. Experienced at Linux/Golang/Mentoring/etc. More information below! A drawing of me coding away. A photo of me giving a talk to a sold-out crowd. Background: Two years ago, I left my job at Red Hat to work on mgmt config full-time. I’m still passionate about this project, and I’m proud of the progress that has been made in the last two years, but it’s time for me to explore new opportunities as well.

Read More

Inexpensive Dynamic DNS over IRC Posted on Sep 1, 2020
Building inexpensive dynamic dns infrastructure for sysadmins


The one thing that I’ve been using for longer than ssh is dynamic dns. In this article I’ll explain how I built an inexpensive, reliable, private alternative to the mainstream tools commonly available today using only golang and IRC. Background: I’ve had an account with dyndns.org for a long time. They used to be free, but then they cut off the free tier, but left existing accounts alone, until they eventually cut all the non-payers off entirely.

Read More

USB Car Audio in a Civic Posted on Aug 24, 2020
Getting USB car audio working in a 2013 Honda Civic EX


Transportation is very important to me. My whole life I’ve never owned a car. I have gotten by with biking, public transit, and various car sharing services. Sadly, biking is incredibly dangerous and stressful, because most cars don’t respect bicycles, and in the winter it’s not very practical, the STM (public transit) offers an incredibly unreliable and unpleasant service, and Car2Go (car sharing) now requires that you use their proprietary smart phone app (you used to be able to just swipe the membership card) so I finally gave up and decided to buy a used car.

Read More

Mgmt Configuration Language: Class and Include Posted on Jul 26, 2019
Class and Include statements in the Mgmt Configuration Language


It’s been a little over a year since I introduced the Mgmt Configuration Language. A lot has happened since then, and I’d like to introduce some of the missing features that weren’t available when the language was first introduced. If you haven’t already read that post, please start there and come back when you’re finished. In this article we’ll learn about classes. Classes: You might remember that the mgmt language called mcl has both statements, and expressions.

Read More

Upcoming Speaking and Events Posted on Feb 2, 2019
Where to get your mgmt fix in the next month


mgmt @ FOSDEM: Later today I’m giving four talks about mgmt at FOSDEM. I’m also giving two more on the following day. (One is a 5 min lightning talk.) I don’t know how I got accepted for so many talks, I just replied to the CFP for every room where I thought I had something interesting to share, and well, here we are! I got rejected for the main track again, but I guess it’s up to me to make my work more generally appealing first.

Read More

A guide to fighting with a Cisco access point Posted on Dec 21, 2018
How eight hours with Cisco Aironet gear made me angry for life


In my hands is a Cisco AIR-CAP-3502-E-A-K9 that I saved from the landfill. After about eight hours of suffering, and what was surely a cancer causing dose of RF-radiation, I now know: 1) How to convince this access point to provide me with wifi. 2) Why Cisco products are probably best left in the garbage. What follows is my setup guide and review for using a Cisco Aironet device. This guide will likely be useful with other models of Cisco devices, but use your own best judgment.

Read More

The End of a Red Giant Posted on Oct 30, 2018
Thoughts on IBM's acquisition of Red Hat


As I’m sure you’ve now heard, Red Hat is being acquired by IBM. Reactions have ranged from: The first two emotions everyone experienced.to: The classic humourous response.to: The classy response.to: The thing we're all curious about.to everything in between. If someone were to leak me a copy of the memo-list 2018-October.txt.gz mailman archive, I’d sure love to see that! (Of course I’m joking and I wouldn’t actually ask anyone to do this.

Read More

Hyperlinks in GNOME terminal Posted on Jun 29, 2018
how to benefit from fancy hyperlinks in your terminal...


Over the years I’ve learned about many of the advantages of using a modern terminal and shell. I’m talking about using bash with GNOME terminal on a modern GNU/Linux distribution. I particularly like switching between the terminal and GUI applications. It’s now even better. Automatic Hyperlinks: Occasionally, some program running in your shell might output what looked like a link: gnome terminal and vte will automatically add links to text that looks like a web hyperlink or an email addressThey can be seen when you hover over them, and a right-click will display a useful context menu:

Read More

Breaking out of the Slack walled garden Posted on Jun 22, 2018
how to use Slack via IRC since the gateway closed...


I’m old school cool. Real hackers chat on open, distributed platforms. Most technical discussion can be found on the Libera.chat IRC network. It’s not perfect, but the advantages clearly outweigh the drawbacks. Recently, I needed to join an existing large “community” on the centralized, proprietary walled garden that is the Slack network. The Problem: Connecting to the the Slack server requires that you use either the proprietary client or their proprietary web app.

Read More

A new sync primitive in golang Posted on May 20, 2018
a new synchronization primitive for golang...


I’ve been working on lots of new stuff in mgmt and I had a synchronization problem that needed solving… Long story short, I built it into a piece of re-usable functionality, exactly like you might find in the sync package. For details and examples, please continue reading… The Problem: I want to multicast a signal to an arbitrary number of goroutines. As you might already know, this can already be done with a chan struct{}.

Read More

Leaving Red Hat Posted on Apr 4, 2018
purpleidea is moving on...


I’ve spent about four years at Red Hat, and now it’s time to move on… TL;DR: had to leave Red Hat and start Patreon to fund mgmt. What follows is a bit of historical rambling, and some forward looking statements. Retrospective: Long-time readers of my blog will know that I was very active in the puppet ecosystem for many years. I learned a lot while writing puppet code, and while building some of my outrageous puppet hacks.

Read More

Running `make` from anywhere Posted on Mar 10, 2018
Run commands from your `Makefile`, even if you're nested deeply!


Sometimes while I’m deep inside mgmt’s project directory, I want to run an operation from the Makefile which lives in the root! Unfortunately, if you do so while nested, you’ll just get: james@computer:~/code/mgmt/resources$ make build make: *** No rule to make target 'build'. Stop. The Ten Minute Solution: I figured I’d hack out a quick solution. What I came up with looks like this: #!/bin/bash # James Shubin, 2018 # run `make` in the first directory (or its parent recursively) that it works in MF='Makefile' # looks for this file, could look for others, but that's silly CWD=$(pwd) # starting here while true; do if [ -e "$MF" ]; then make $@ # run make!

Read More

Mgmt Configuration Language Posted on Feb 5, 2018
Introducing the Mgmt Configuration Language


Mgmt has given us a lot of great features, demos, and insight into how infrastructure automation should be managed. Unfortunately it wasn’t a complete tool yet, because it was missing a crucial piece for its completeness. That all changes today with the addition of: the language. An ideal language allows us to express easily what is useful for the programming task, and at the same time makes it difficult to write what leads to incomprehensible or incorrect programs.

Read More

Hello World! Posted on Nov 13, 2017
First test of new blog


Hello! This is a new version of my website and blog. If you experience any issues, please let me know! Happy Hacking, James You can follow James on Mastodon for more frequent updates and other random thoughts. You can follow James on Twitter for more frequent updates and other random thoughts. You can support James on GitHub if you'd like to help sustain this kind of content. You can support James on Patreon if you'd like to help sustain this kind of content.

Read More

Copyleft is Dead. Long live Copyleft! Posted on Oct 17, 2017


As you may have noticed, we recently re-licensed mgmt from the AGPL (Affero General Public License) to the regular GPL. This is a post explaining the decision and which hopefully includes some insights at the intersection of technology and legal issues. Disclaimer: I am not a lawyer, and these are not necessarily the opinions of my employer. I think I’m knowledgeable in this area, but I’m happy to be corrected in the comments.

Read More

Extracting movies from libreoffice Posted on Jul 12, 2017


I have a short movie that I imported into a libreoffice presentation. I wanted a copy of that movie back, but I couldn’t figure out how to extract a copy. In desperation, I figured I’d try opening the file with file-roller, the GNOME archive manager. james@computer:/tmp$ file mgmt-berlin-osdc-17may2017.odp mgmt-berlin-osdc-17may2017.odp: OpenDocument Presentation james@computer:/tmp$ mkdir out james@computer:/tmp$ file-roller -f mgmt-berlin-osdc-17may2017.odp -e out/ [snip] james@computer:/tmp$ cd out/ james@computer:/tmp/out$ ls Configurations2/ Media/ meta.xml Pictures/ styles.

Read More

Declarative vs. Imperative paradigms Posted on May 5, 2017


Recently, while operating two different remote-controlled appliances, I realized that it was high time for a discussion about declarative and imperative paradigms. Let’s start by looking at the two remotes: Two different “remotes”. The one on the left operates a television, and the one on the right controls a central heating and cooling system.At first glance you will notice that one of these remotes is dark, and the other is light.

Read More

Osmocom femtocell un-boxing Posted on Apr 30, 2017


LaForge and the fine folks at Osmocom (Sysmocom) recently had a femtocell giveaway. I didn’t expect to have much time to hack on things, but they were still quite generous in sending me one. It arrived, and I took some un-boxing photos for anyone who is curious. A box arrived in the mail…Which recurses into an inner box…Inner box is box like.Finally… The unit is displayed.Here it is in all its glory.

Read More

Metaparameters in mgmt Posted on Mar 1, 2017


In mgmt we have meta parameters. They are similar in concept to what you might be familiar with from other tools, except that they are more clearly defined (in a single struct) and vastly more powerful. In mgmt, a meta parameter is a parameter which is codified entirely in the engine, and which can be used by any resource. In contrast with Puppet, require/before are considered meta parameters, whereas in mgmt, the equivalent is a graph edge, which is not a meta parameter.

Read More

Faster golang builds Posted on Feb 26, 2017


I’ve been hacking in golang since before version 1.4, and the speed at which my builds finished has been mostly trending downwards. Let’s look into the reasons and some fixes. TL;DR click-bait title: “Get 4x faster golang builds with this one trick!”. Here are the three reasons my builds got slower: The compiler Before version 1.5, the compiler was written in C but with that release, it moved to being pure golang.

Read More

Ten minute hacks: Process pause & resume Posted on Jan 6, 2017


I’m old school and still rocking an old X220 laptop because I didn’t like the new ones. My battery life isn’t as great as I’d like it to be, but it gets worse when some “webapp” (which I’d much rather have as a native GTK+ app) causes Firefox to rev my CPU with their websocket (hi gmail!) poller. This seems to happen most often on planes or when I’m disconnected from the internet.

Read More

Send/Recv in mgmt Posted on Dec 7, 2016


I previously published “A revisionist history of configuration management”. I meant for that to be the intro to this article, but it ended up being long enough that it deserved a separate post. I will explain Send/Recv in this article, but first a few clarifications to the aforementioned article. Clarifications I mentioned that my “revisionist history” was inaccurate, but I failed to mention that it was also not exhaustive! Many things were left out either because they were proprietary, niche, not well-known, of obscure design or simply for brevity.

Read More

A revisionist history of configuration management Posted on Nov 30, 2016


I’ve got a brand new core feature in mgmt called send/recv which I plan to show you shortly, but first I’d like to start with some background. History This is my historical perspective and interpretation about the last twenty years in configuration management. It’s likely inaccurate and slightly revisionist, but it should be correct enough to tell the design story that I want to share. Sometime after people started to realize that writing bash scripts wasn’t a safe, scalable, or reusable way to automate systems, CFEngine burst onto the scene with the first real solution to this problem.

Read More

Remote execution in mgmt Posted on Oct 7, 2016


Bootstrapping a cluster from your laptop, or managing machines without needing to first setup a separate config management infrastructure are both very reasonable and fundamental asks. I was particularly inspired by Ansible’s agent-less remote execution model, but never wanted to build a centralized orchestrator. I soon realized that I could have my ice cream and eat it too. Prior knowledge If you haven’t read the earlier articles about mgmt, then I recommend you start with those, and then come back here.

Read More

mgmt has a logo Posted on Sep 30, 2016


The mgmt config project got a logo! The full commit is here. Thanks to Sarah Jane Cox for creating it. Happy Hacking, James PS: I might have a few stickers to give out too! Ask me next time you see me if you’d like one! Alternatively, use the artwork to make your own and share with your friends! You can follow James on Mastodon for more frequent updates and other random thoughts.

Read More

Live dmesg following Posted on Aug 29, 2016


All good sysadmins eventually learn about using tail -F to tail files. Yes upper-case F is superior. Around the time I wrote that article, I remember wanting to stream dmesg output too! The functionality wasn’t available without some sort of polling hack, but it turns out that kernel support for this actually landed around the same time in version 3.5.0! Most GNU/Linux distros are probably running a new enough version by now, and you can now dmesg –follow (or dmesg -w):

Read More

Seen in downtown Montreal... Posted on Aug 3, 2016


The Technical Blog of James was seen on an outdoor electronic display in downtown Montreal! Thanks to one of my readers for sending this in. I guess the smart phone revolution is over, and people are taking to reading my articles on bigger screens! The “poutine” is decent proof that this is probably Montreal.If you’ve got access to a large electronic display, put up the blog, snap a photo, and send it my way!

Read More

Ten minute hacks: Hacking airplane headphones Posted on Jul 18, 2016


I was stuck on a 14 hour flight last week, and to my disappointment, only one of the two headphone speakers were working. The plane’s media centre has an audio connector that looks like this: Someone should consider probing this USB port.The hole to the left is smaller than a 3.5mm headphone jack, and designed for a proprietary headphone connector that I didn’t have, and the two holes to the right are part of a different proprietary connector which match with the cheap airline headphones to provide the left and right audio channels.

Read More

Automatic clustering in mgmt Posted on Jun 20, 2016


In mgmt, deploying and managing your clustered config management infrastructure needs to be as automatic as the infrastructure you’re using mgmt to manage. With mgmt, instead of a centralized data store, we function as a distributed system, built on top of etcd and the raft protocol. In this article, I’ll cover how this feature works. Foreword: Mgmt is a next generation configuration management project. If you haven’t heard of it yet, or you don’t remember why we use a distributed database, start by reading the previous articles:

Read More

Upcoming speaking In Hong Kong and South Africa Posted on Jun 20, 2016


I’m thrilled to tell you that I’ll be speaking about mgmt in Hong Kong and South Africa. It will be my first time to both countries and my first time to Asia and Africa! In Hong Kong I’ll be speaking at HKOSCon2016. In South Africa I’ll be speaking at DebConf16. I’m looking forward to meeting with many of the hard-working Debian hackers, and collaborating with them to build and promote excellent Free Software.

Read More

One hour hacks: Remote LUKS over SSH Posted on Apr 25, 2016


I have a GNU/Linux server which I mount a few LUKS encrypted drives on. I only ever interact with the server over SSH, and I never want to keep the LUKS credentials on the remote server. I don’t have anything especially sensitive on the drives, but I think it’s a good security practice to encrypt it all, if only to add noise into the system and for solidarity with those who harbour much more sensitive data.

Read More

Automatic grouping in mgmt Posted on Mar 30, 2016


In this post, I’ll tell you about the recently released “automatic grouping” or “AutoGroup” feature in mgmt, a next generation configuration management prototype. If you aren’t already familiar with mgmt, I’d recommend you start by reading the introductory post, and the second post. There’s also an introductory video. Resources in a graph Most configuration management systems use something called a directed acyclic graph, or DAG. This is a fancy way of saying that it is a bunch of circles (vertices) which are connected with arrows (edges).

Read More

Automatic edges in mgmt Posted on Mar 14, 2016


It’s been two months since I announced mgmt, and now it’s time to continue the story by telling you more about the design of what’s now in git master. Before I get into those details, let me quickly recap what’s happened since then. Mgmt community recap: I gave the first public presentation about mgmt at CfgMgmtCamp. I repeated the talk at DevConf.cz. The video recording is available. Felix wrote about his work cross compiling puppet code to mgmt.

Read More

Introducing: git tpush Posted on Feb 16, 2016


On today’s issue of “one hour hacks”, I’ll show you how you can stop your git drive-by’s to git master from breaking your CI tests… Let’s continue! The problem: Sometimes I’ve got a shitty one-line patch that I want to push to git master. I’m usually right, and everything tests out fine, but usually isn’t always, and then I look silly while I frantically try to fix git master on a project that I maintain.

Read More

Debugging golang programs Posted on Feb 15, 2016


I’ve been writing a lot of golang lately. I’ve hit painful problems in the past. Here are some debugging tips. Hopefully they help you out. I bet you don’t know #2. #0 Use log.Printf: This should go without saying, but I’m ashamed to say it’s what I use the most. We’ve only been C programming for 44+ years, and it’s still what is most useful! #1 Use go run -race:

Read More

Next generation configuration mgmt Posted on Jan 18, 2016


It’s no secret to the readers of this blog that I’ve been active in the configuration management space for some time. I owe most of my knowledge to what I’ve learned while working with Puppet and from other hackers working in and around various other communities. I’ve published, a number, of articles, in an, attempt, to push, the field, forwards, and to, share the, knowledge, that I’ve, learned, with others. I’ve spent many nights thinking about these problems, but it is not without some chagrin that I realized that the current state-of-the-art in configuration management cannot easily (or elegantly) solve all the problems for which I wish to write solutions.

Read More

Upcoming speaking Posted on Jan 18, 2016


I’ve got a few upcoming speaking engagements. If you’ll be attending one of these events, come see me or any of the other excellent speakers! Please remember to check the official schedules in case there are any changes! I’ll be speaking at the Brussels CentOS Dojo: Automated Infrastructure Testing with Oh-My-Vagrant …and the CentOS CI Time/date unconfirmed: I’ll be showing some CI tricks, and showing you how the CentOS CI is the perfect CI for multi-machine test environments.

Read More

Trying out Ceph with Oh-My-Vagrant Posted on Dec 28, 2015


Daniel P. Berrangé wrote about trying out a single node ceph cluster. I decided to take his article and turn it into an Oh-My-Vagrant omv.yaml file. It took me about two minutes to do so, and two hours to debug a problem caused by something I had broken on my laptop. If you’d like to replicate his article in less than 5 minutes, pull down the omv.yaml file that I’ve just published and run omv up.

Read More

Matching arbitrary URL's to custom Firefox profiles Posted on Nov 14, 2015


We’re constantly clicking on all sorts of different URL’s throughout the day. These clickable links appear in webpages (including in “web apps” like gmail) in mail clients like Evolution, in terminals such as GNOME-terminal, and any other GTK+ app on your GNU/Linux desktop. I wanted to perform custom actions when arbitrary URL’s are clicked, including running certain links in separate Firefox profiles. There are a bunch of different steps you have to do to get this working, but it should be easy to follow along.

Read More

Thanking Oh-My-Vagrant contributors for version 1.0.0 Posted on Aug 18, 2015


The Oh-My-Vagrant project became public about one year ago and at the time it was more of a fancy template than a robust project, but 188 commits (and counting) later, it has gotten surprisingly useful and mature. james@computer:~/code/oh-my-vagrant$ git rev-list HEAD --count 188 james@computer:~/code/oh-my-vagrant$ git log $(git log --pretty=format:%H|tail -1) commit 4faa6c89cce01c62130ef5a6d5fa0fff833da371 Author: James Shubin <[email protected]> Date: Thu Aug 28 01:08:03 2014 -0400 Initial commit of vagrant-puppet-docker-template... This is an attempt to prototype a default environment for vagrant+puppet+docker hacking.

Read More

Vagrant and Oh-My-Vagrant on RHEL7 Posted on Aug 11, 2015


My employer keeps paying me, which I appreciate, so it’s good to spend some time to make sure RHEL7 customers get a great developer experience! So here’s how to make vagrant, vagrant-libvirt and Oh-My-Vagrant work on RHEL 7+. The same steps should work for CentOS 7+. I’ll first paste the commands you need to run, and then I’ll explain what’s happening for those that are interested: # run these commands, and then get hacking!

Read More

Making an empty RPM Posted on Aug 11, 2015


I am definitely not an RPM expert, in fact, I’m afraid of it, but with recent tools such as COPR, and my glorious Makefile, some aspects of it have become palatable. This post will be about a recent journey I had building the most useless RPM ever. A video of my journey building this RPM.Because of reasons, I wanted to satisfy an RPM dependency for a package that I wanted to install without rebuilding that RPM.

Read More

Golang parallelism issues causing “too many open files” error Posted on Jul 27, 2015


I’ve been hacking in golang for a while, but I’ll admit that I didn’t get too deep into some of the language nuances until more recently. Since some of them have started to bite me, here’s a little post-mortem of one of the problems I was having. After hacking and testing code all day, I made a seemingly innocuous change, and when running my program, I saw the following error:

Read More

Git archive with submodules and tar magic Posted on Jul 23, 2015


Git submodules are actually a very beautiful thing. You might prefer the word powerful or elegant, but that’s not the point. The downside is that they are sometimes misused, so as always, use with care. I’ve used them in projects like puppet-gluster, oh-my-vagrant, and others. If you’re not familiar with them, do a bit of reading and come back later, I’ll wait. I recently did some work packaging Oh-My-Vagrant as RPM’s.

Read More

Oh-My-Vagrant “Mainstream” mode and COPR RPM's Posted on Jul 8, 2015


Making Oh-My-Vagrant (OMV) more developer accessible and easy to install (from a distribution package like RPM) has always been a goal, but was previously never a priority. This is all sorted out now. In this article, I’ll explain how “mainstream” mode works, and how the RPM work was done. (I promise this will be somewhat interesting!) Prerequisites: If you haven’t read any of the previous articles about Oh-My-Vagrant, I’d recommend you start there.

Read More

A super privileged Puppet container Posted on Jun 12, 2015


In this new crazy world of containers and immutable hosts, one might still want to run previous generation software such as Puppet on a current generation Atomic host. This article will explain how you can do that, and offer some proof of concept code. The atomic host doesn’t provide a yum or dnf command, because the software is pre-baked into a read-only /usr/ partition. To “install” (to use) additional software, it usually needs to be distributed and run as a container.

Read More

Kubernetes clusters with Oh-My-Vagrant Posted on May 2, 2015


I’ve added the ability to deploy a Kubernetes cluster with Oh-My-Vagrant (omv). I’ve also built an automated developer experience so that you can test your Kubernetes powered app in minutes. If you want to redeploy a new version, or see how your app behaves during a rolling update, you can use omv to test this out in minutes! I’ve recorded a screencast (~15 min), if you’d like to see some of this in action.

Read More

Docker containers in Oh-My-Vagrant Posted on Apr 20, 2015


The Oh-My-Vagrant (omv) project is an easy way to bootstrap a development environment. It is particularly useful for spinning up an arbitrary number of virtual machines in Vagrant without writing ruby code. For multi-machine container development, omv can be used to help this happen more naturally. Oh-My-Vagrant can be very useful as a docker application development environment. I’ve made a quick (<9min) screencast demoing this topic. Please have a look:

Read More

Sharing dev environments with Oh-My-Vagrant Posted on Apr 8, 2015


With Oh-My-Vagrant (omv) you can set up a dev environment in seconds. (Read the omv introduction if you’ve never used it before!) Since everything is defined in a single omv.yaml file, it is easy to share your cluster prototype with a friend! The one missing feature was associating code with this config file. This is now possible! Let me show you how it works… In the omv.yaml file there is an extern variable.

Read More

Fancy git aliases and git cherryfetch Posted on Mar 16, 2015


Here are two quick git tricks that I’ve added to my toolbox lately… I wanted to create a git alias that takes in argv from the command, but in the middle of the command. Here’s the hack that I came up with for the [alias] section of my ~/.gitconfig: [alias] # cherryfetch fetches a repo ($1) / branch ($2) and applies it rebased! # the && true at the end eats up the appended args cherryfetch = !

Read More

Building RHEL Vagrant Boxes with Vagrant-Builder Posted on Feb 23, 2015


Vagrant is a great tool for development, but Red Hat Enterprise Linux (RHEL) customers have typically been left out, because it has been impossible to get RHEL boxes! It would be extremely elegant if hackers could quickly test and prototype their code on the same OS as they’re running in production. Secondly, when hacking on projects that have a long initial setup phase (eg: a long rpm install) it would be excellent if hackers could roll their own modified base boxes, so that certain common operations could be re-factored out into the base image.

Read More

Introducing: Silent counter Posted on Feb 9, 2015


You might want to write code that can tell how many iterations have passed since some action occurred. Alternatively, you might want to know if it’s the first time a machine has run Puppet. To do these types of things, you might wish to have a monotonically increasing counter in your Puppet manifest. Since one did not exist, I set out to build one! The code: If you just want to try the code, and skip the ramble, you can include common::counter into your manifest.

Read More

Replying to mailing lists with Evolution Posted on Nov 27, 2014


I use the Evolution mail client. It does have a few annoying bugs, but it has a plethora of great features too! Hopefully this post will inspire you to help hack on this piece of software and fix the bugs! Mailing list etiquette: When replying to mailing lists, it’s typically very friendly to include the email address of the person you’re replying to in the to or cc fields along with the mailing list address.

Read More

Captive web portals are considered harmful Posted on Nov 27, 2014


Recently, when I tried to access http://slashdot.org/ in Firefox, I would see my browser title bar flash briefly to “AT&T GUI”, and then I would get redirected to: http://slashdot.org/cgi-bin/redirect.ha which returns slashdot’s custom error 404 page! What is going on? (Read on for answer…) Did slashdot mess up their mod_rewrite config? (Nope, works fine in a different browser…) Did my HTTPS everywhere extension go crazy? (Nope, still broken when disabled…) Are my HTTP requests being MITM-ed?

Read More

The switch as an ordinary GNU/Linux server Posted on Nov 4, 2014


The fact that we manage the switches in our data centres differently than any other server is patently absurd, but we do so because we want to harness the power of a tiny bit of silicon which happens to be able to dramatically speed up the switching bandwidth. beware of proprietary silicon, it’s absurd!That tiny bit of silicon is known as an ASIC, or an application specific integrated circuit, and one particularly well performing ASIC (which is present in many commercially available switches) is called the Trident.

Read More

Testing Evolution's git master and GNOME continuous Posted on Oct 22, 2014


I’ve wanted a feature in Evolution for a while. It was formally requested in 2002, and it just recently got fixed in git master. I only started publicly groaning about this missing feature in 2013, and mcrha finally patched it. I tested the feature and found a small bug, mcrha patched that too, and I finally re-tested it. Now I’m blogging about this process so that you can get involved too!

Read More

Hacking out an Openshift app Posted on Oct 18, 2014


I had an itch to scratch, and I wanted to get a bit more familiar with Openshift. I had used it in the past, but it was time to have another go. The app and the code are now available. Feel free to check out: https://pdfdoc-purpleidea.rhcloud.com/ This is a simple app that takes the URL of a markdown file on GitHub, and outputs a pandoc converted PDF. I wanted to use pandoc specifically, because it produces PDF’s that were beautifully created with LaTeX.

Read More

Continuous integration for Puppet modules Posted on Oct 10, 2014


I just patched puppet-gluster and puppet-ipa to bring their infrastructure up to date with the current state of affairs… What’s new? Better README’s Rake syntax checking (fewer oopsies) CI (testing) with travis on git push (automatic testing for everyone) Use of .pmtignore to ignore files from puppet module packages (finally) Pushing modules to the forge with blacksmith (sweet!) This last point deserves another mention. Puppetlabs created the “forge” to try to provide some sort of added value to their stewardship.

Read More

Fixing dropbox “conflicted copy” problems Posted on Sep 30, 2014


I usually avoid proprietary cloud services because of freedom, privacy and vendor lock-in concerns. In addition, there are some excellent libre (and hosted) services such as WordPress, Wikipedia and OpenShift which don’t have the above problems. Thirdly, there are every day Free Software tools such as Fedora GNU/Linux, Libreoffice, and git-annex-assistant which make my computing much more powerful. Finally, there are some hosted services that I use that don’t lock me in because I use them as push-only mirrors, and I only interact with them using Free Software tools.

Read More

Introducing: Oh My Vagrant! Posted on Sep 3, 2014


If you’re a reader of my code or of this blog, it’s no secret that I hack on a lot of puppet and vagrant. Recently I’ve fooled around with a bit of docker, too. I realized that the vagrant, environments I built for puppet-gluster and puppet-ipa needed to be generalized, and they needed new features too. Therefore… Introducing: Oh My Vagrant! Oh My Vagrant is an attempt to provide an easy to use development environment so that you can be up and hacking quickly, and focusing on the real devops problems.

Read More

Rough data density calculations Posted on Aug 27, 2014


Seagate has just publicly announced 8TB HDD’s in a 3.5” form factor. I decided to do some rough calculations to understand the density a bit better… Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since I always work in base 2, but I hate the -bi naming conventions. Seagate is most likely announcing an 8TB HDD, which is actually smaller than a true 8TiB drive.

Read More

Hybrid management of FreeIPA types with Puppet Posted on Jul 24, 2014


(Note: this hybrid management technique is being demonstrated in the puppet-ipa module for FreeIPA, but the idea could be used for other modules and scenarios too. See below for some use cases…) The error message that puppet hackers are probably most familiar is: Error: Duplicate declaration: Thing[/foo/bar] is already declared in file /tmp/baz.pp:2; cannot redeclare at /tmp/baz.pp:4 on node computer.example.com Typically this means that there is either a bug in your code, or someone has defined something more than once.

Read More

One minute hacks: the nautilus scripts folder Posted on Jun 26, 2014


Master SDN hacker Flavio sent me some tunes. They were sitting on my desktop in a folder: $ ls ~/Desktop/ uncopyrighted_tunes_from_flavio/ I wanted to listen them while hacking, but what was the easiest way…? I wanted to use the nautilus file browser to select which folder to play, and the totem music/video player to do the playing. Drop a file named totem into: ~/.local/share/nautilus/scripts/ with the contents: #!/bin/bash # o hai from purpleidea exec totem -- "$@" and make it executable with:

Read More

Securely managing secrets for FreeIPA with Puppet Posted on Jun 6, 2014


Configuration management is an essential part of securing your infrastructure because it can make sure that it is set up correctly. It is essential that configuration management only enhance security, and not weaken it. Unfortunately, the status-quo of secret management in puppet is pretty poor. In the worst (and most common) case, plain text passwords are found in manifests. If the module author tried harder, sometimes these password strings are pre-hashed (and sometimes salted) and fed directly into the consumer.

Read More

Hiera data in modules and OS independent puppet Posted on Jun 4, 2014


Earlier this year, R.I.Pienaar released his brilliant data in modules hack, a few months ago, I got the chance to start implementing it in Puppet-Gluster, and today I have found the time to blog about it. What is it? R.I.’s hack lets you store hiera data inside a puppet module. This can have many uses including letting you throw out the nested mess that is commonly params.pp, and replace it with something file based that is elegant and hierarchical.

Read More

Restarting GNOME shell via SSH Posted on May 29, 2014


When GNOME shell breaks, you get to keep both pieces. The nice thing about shell failures in GNOME 3, is that if you’re able to do a restart, the active windows are mostly not disturbed. The common way to do this is to type ALT-F2, r, <ENTER>. Unfortunately, you can’t always type that in if your shell is very borked. If you are lucky enough to have SSH access, and another machine, you can log in remotely and run this script:

Read More

Vagrant on Fedora with libvirt (reprise) Posted on May 13, 2014


Vagrant has become the de facto tool for devops. Faster iterations, clean environments, and less overhead. This isn’t an article about why you should use Vagrant. This is an article about how to get up and running with Vagrant on Fedora with libvirt easily! Background: This article is an update of my original Vagrant on Fedora with libvirt article. There is still lots of good information in that article, but this one should be easier to follow and uses updated versions of Vagrant and vagrant-libvirt.

Read More

Keeping git submodules in sync with your branches Posted on May 6, 2014


This is a quick trick for making working with git submodules more magic. One day you might find that using git submodules is needed for your project. It’s probably not necessary for everyday hacking, but if you’re glue-ing things together, it can be quite useful. Puppet-Gluster uses this technique to easily include all the dependencies needed for a Puppet-Gluster+Vagrant automatic deployment. If you’re a good hacker, you develop things in separate feature branches.

Read More

Working at RedHat Posted on Apr 2, 2014


So this happened: James just James at RedHat headquarters in North Carolina wearing his new red hat.RedHat made me an offer, and I am happy to say that I have just started this week! I am proud to have joined a company that employs many of the worlds foremost, highly professional and clever hackers. It is indubitably the best Free Software [1] / Open Source company out there, and they ship some of the greatest and most elegant software available.

Read More

Puppet-Gluster now available as RPM Posted on Mar 27, 2014


I’ve been afraid of RPM and package maintaining [1] for years, but thanks to Kaleb Keithley, I have finally made some RPM’s that weren’t generated from a high level tool. Now that I have the boilerplate done, it’s a relatively painless process! In case you don’t know kkeithley, he is a wizard [2] who happens to also be especially cool and hardworking. If you meet him, be sure to buy him a $BEVERAGE.

Read More

Introducing Puppet Exec['again'] Posted on Mar 24, 2014


Puppet is missing a number of much-needed features. That’s the bad news. The good news is that I’ve been able to write some of these as modules that don’t need to change the Puppet core! This is an article about one of these features. Posit: It’s not possible to apply all of your Puppet manifests in a single run. I believe that this holds true for the current implementation of Puppet.

Read More

Preserving your working directory in gnome-terminal Posted on Mar 20, 2014


I use gnome-terminal for most of my hacking. In fact, I use it so much, that I’ll often have multiple tabs open for a particular project. Here’s my workflow: Control+Alt+t (My shortcut to open a new gnome-terminal window.) cd ~/code/some_cool_hack/ # directory of some cool hack Control-Shift-t (Shortcut to open a new gnome-terminal tab.) Hack, hack, hack… The problem is that the new tab that I’ve created will have a $PWD of ~, instead of keeping the $PWD of ~/code/some_cool_hack/, which is the project I’m working on!

Read More

Speaking at SCALE today! Posted on Feb 21, 2014


I’ll be giving a talk at SCALE today about automatically deploying GlusterFS with Puppet-Gluster and Vagrant. I’ll be giving some live demos, and this will cover some of the material from: Automatically deploying GlusterFS with Puppet-Gluster + Vagrant! and it will contain excerpts from: Screencasts of Puppet-Gluster + Vagrant I’ll also be talking about some new upcoming features, and am happy to answer all of your questions! The talk will be part of Infrastructure.

Read More

Building a snow shelter Posted on Feb 17, 2014


To give you a break from the usual GNU/Linux/DevOps/Puppet/GlusterFS drab, I’ve decided to have a go at writing a different kind of technical article. This article will show you how to build the traditional Canadian snow dwelling known as a quinzee. If you will be travelling to Canada, I recommended that you read through this article ahead of time, so that you don’t offend your host by being unfamiliar with their traditional living accommodations.

Read More

Scathing review of the Lenovo X240 Posted on Feb 2, 2014


I’m using a Lenovo X201 with 8GiB of RAM. Apart from some minor issues, I’ve been very satisfied with this laptop. It’s over four years old, and so I decided to see what’s available on the horizon. I did not buy an X240 because of the following reasons: The X240 has only one slot for RAM and thus supports a maximum of 8GiB. I think it’s pretty ridiculous for any successor to the X230 to support less RAM.

Read More

Show the exit status in your $PS1 Posted on Jan 29, 2014


As an update to my earlier article, a friend gave me an idea of how to make my $PS1 even better… First, the relevant part of my ~/.bashrc: ps1_prompt() { local ps1_exit=$? if [ $ps1_exit -eq 0 ]; then #ps1_status=`echo -e "\[\033[32m\]"'\$'"\[\033[0m\]"` ps1_status='\$' else ps1_status=`echo -e "\[\033[1;31m\]"'\$'"\[\033[0m\]"` fi ps1_git='' if [ "$(__git_ps1 %s)" != '' -a "$(__git_ps1 %s)" != 'master' ]; then ps1_git=" (\[\033[32m\]"$(__git_ps1 "%s")"\[\033[0m\])" fi PS1="${debian_chroot:+($debian_chroot)}\u@\h:\[\033[01;34m\]\w\[\033[00m\]${ps1_git}${ps1_status}" } # preserve earlier PROMPT_COMMAND entries.

Read More

Screencasts of Puppet-Gluster + Vagrant Posted on Jan 27, 2014


I decided to record some screencasts to show how easy it is to deploy GlusterFS using Puppet-Gluster+Vagrant. You can follow along even if you don’t know anything about Puppet or Vagrant. The hardest part of this process was producing the actual videos! If recommend first reading my earlier articles if you’re planning on following along: Vagrant on Fedora with libvirt Vagrant vsftp and other tricks Vagrant clustered SSH and ‘screen’ Automatically deploying GlusterFS with Puppet-Gluster + Vagrant!

Read More

Building base images for Vagrant with a Makefile Posted on Jan 20, 2014


I needed a base image “box” for my Puppet-Gluster+Vagrant work. It would have been great if good boxes already existed, and even better if it were easy to build my own. As it turns out, I wasn’t able to satisfy either of these conditions, so I’ve had to build one myself! I’ve published all of my code, so that you can use these techniques and tools too! Status quo: Having an NIH problem is bad for your vision, and it’s best to benefit from existing tools before creating your own.

Read More

Testing GlusterFS during “Glusterfest” Posted on Jan 16, 2014


The GlusterFS community is having a “test day”. Puppet-Gluster+Vagrant is a great tool to help with this, and it has now been patched to support alpha, beta, qa, and rc releases! Because it was built so well (*cough*, shameless plug), it only took one patch. Okay, first make sure that your Puppet-Gluster+Vagrant setup is working properly. I have only tested this on Fedora 20. Please read: Automatically deploying GlusterFS with Puppet-Gluster+Vagrant!

Read More

Automatically deploying GlusterFS with Puppet-Gluster + Vagrant! Posted on Jan 8, 2014


Puppet-Gluster was always about automating the deployment of GlusterFS. Getting your own Puppet server and the associated infrastructure running was never included “out of the box”. Today, it is! (This is big news!) I’ve used Vagrant to automatically build these GlusterFS clusters. I’ve tested this with Fedora 20, and vagrant-libvirt. This won’t work with Fedora 19 because of bz#876541. I recommend first reading my earlier articles for Vagrant and Fedora: Vagrant on Fedora with libvirt Vagrant vsftp and other tricks Vagrant clustered SSH and ‘screen’ Once you’re comfortable with the material in the above articles, we can continue…

Read More

Vagrant clustered SSH and ‘screen’ Posted on Jan 2, 2014


Some fun updates for vagrant hackers… I wanted to use the venerable clustered SSH (cssh) and screen with vagrant. I decided to expand on my vsftp script. First read: Vagrant on Fedora with libvirt and Vagrant vsftp and other tricks to get up to speed on the background information. Vagrant screen: First, a simple screen hack… I often use my vssh alias to quickly ssh into a machine, but I don’t want to have to waste time with sudo-ing to root and then running screen each time.

Read More

Vagrant vsftp and other tricks Posted on Dec 21, 2013


As I previously wrote, I’ve been busy with Vagrant on Fedora with libvirt, and have even been submitting, patches and issues! (This “closed” issue needs solving!) Here are some of the tricks that I’ve used while hacking away. Default provider: I should have mentioned this in my earlier article but I forgot: If you’re always using the same provider, you might want to set it as the default. In my case I’m using vagrant-libvirt.

Read More

Vagrant on Fedora with libvirt Posted on Dec 9, 2013


Apparently lots of people are using Vagrant these days, so I figured I’d try it out. I wanted to get it working on Fedora, and without Virtualbox. This is an intro article on Vagrant, and what I’ve done. I did this on Fedora 19. Feel free to suggest improvements. Intro: Vagrant is a tool that easily provisions virtual machines, and then kicks off a configuration management deployment like Puppet. It’s often used for development.

Read More

Now syndicated on “Planet Fedora” Posted on Dec 9, 2013


I’m now syndicated on the Fedora Project planet. If you haven’t read through my blog yet, let me introduce myself, I’m James, and I write The Technical Blog of James. I’m a sysadmin, DevOps/Puppet hacker, I.T./network architect and physiologist. Hi! I run Fedora as my primary desktop, but I also use it for servers, particularly for development before future versions of RHEL and CentOS release. I’m most well-known for Puppet-Gluster, but I’ve also written a decent Puppet-IPA (FreeIPA) module.

Read More

Advanced recursion and memoization in Puppet Posted on Nov 27, 2013


As a follow-up to my original article on recursion in Puppet, and in my attempt to Push Puppet (to its limit), I’ll now attempt some more advanced recursion techniques in Puppet. In my original recursion example, the type does recurse, but the callee cannot return any value to the caller because it is a type, and not strictly a function. This limitation immediately limits the usefulness of this technique, but I’ll try to press on!

Read More

Documentation for Puppet-Gluster Posted on Nov 20, 2013


Ironically, one of the reasons that I started writing Puppet code, was so that I could spend more time designing and building, and less time writing documentation. I suppose I’m a victim of my success, because Puppet-Gluster has grown large enough to warrant its own documentation. So I gave in, and put together some documentation. It’s available as markdown, and, also as a pdf. As before, there is an examples/ directory which you might want to use as additional reference.

Read More

Iteration in Puppet Posted on Nov 17, 2013


People often ask how to do iteration in Puppet. Most Puppet users have a background in imperative programming, and are already very familiar with for loops. Puppet is sometimes confusing at first, because it is actually (or technically, contains) a declarative, domain-specific language. In general, DSL’s aren’t always Turing complete, nor do they need to support loops, but this doesn’t mean you can’t iterate. Until recently, Puppet didn’t have an explicit looping construct, and it is quite possible to build complex modules without using this new construct.

Read More

Pushing Puppet at Puppet Camp DC, LISA 2013 Posted on Nov 5, 2013


Hi there, I hope you enjoyed my “Pushing Puppet (to its limit)” talk and demos from Puppet Camp D.C., LISA 2013. As requested, I’ve posted the code and slides. Here is the code: https://github.com/purpleidea/puppet-pushing This module will require three modules as dependencies. The dependencies are: My Puppet-Common module https://github.com/purpleidea/puppet-common My Puppet-Runonce module https://github.com/purpleidea/puppet-runonce My Puppet-FSM module https://github.com/purpleidea/puppet-fsm Each example doesn’t require all the dependencies, so if you’re only interested in the FSM, you only need that module.

Read More

Links...

M