There is no cheaper storage than USB sticks. When writing this, they’re 30 bucks for 64GB. Unlike hard drives, they don’t eat a lot of juice when you keep them plugged in, they’re quiet, they are not sensitive to having a dusty home (thank god!), and they last. It’s also very easy to wipe out the dumb cruftware that comes pre-installed on them.

If you’re a command line guy like me, it’s obvious that you must have a Raspberry Pi server at home. It’s also obvious that you need to cram it full as many USB sticks as you can, in order to thumb your paranoid nose at Dropbox. Before pointing said nose at a full screen terminal and punching in powerful, arcane commands that connect you to the fathers of Unix. And feel like them. Even though you have a mouse, copy-and-paste, and command completion.

What gets really, really annoying is having your assorted drives from different eras and sizes on different mount points. Enter LVM.

When you get your next bigger thumb drive and cram it into your pi, don’t do mkfs /dev/sdx1 right away. (/dev/sdx1 is the placeholder device name used in this post for your new usb drive, which assumes you have already connected 23 thumb drives to your pi, which is the only reasonable thing to do. If you’re new to this, your actual device name could also be /dev/sda1 or /dev/sdb1).

So instead of doing

right away, you do:

So far this is just typing extra commands, but now you can transfer the data of one of your existing 23 usb sticks into /mnt/store.

And admit it to the group

Et voila! Instead of two annoyingly invidual mount points with 30G and 60G capacity, you have one with 90G capacity!

This means, whenever you run out of space, instead of extending your dropbox plan, you just add another thumb drive, haze him, and admit him to your group.

Of course you need some data security, so theoretically you should use half of your usb drive army to create a software raid1, and then pile your LVM on top of that.

That’s where git annex does a world of good.

Like with any decent dropbox replacement, you will probably have all those files on your laptop, your server, your home laptop, and the server in the basement you forgot about. That is more than two copies, and that is a good thing. You also get checksums.

So if you put your git annex on your LVM, you will have a pool of checksummed extendible storage that you can sync to any command line you come across.

And if you care about someone enough to bother, and that person has a recent mac, you can set up the same thing with git annex assistant.

Nice.

Here’s a hard one… I’m getting a hunch Carlo Capocasa is not going to click it. To long. To unwieldy. Everyone thinks I’m in the Mafia.

c2c would have been great, just my initials and the coast to coast reference. But then there’s that band from France. Bummer.

So I settled for c4e, which is short for create (c, four letters, and then e). It’s what I stand for. Create, don’t worry about getting or moving around. Create. The world will be better.

Note: This stuff works on Ubuntu 12.04, Precise pangolin. It should be fairly easy to port to other distributions.

There has been a recent kernel bug that has, on certain systems like my wonderful, wonderful Lenovo y560p , lead to the massive uprising of interrupts, which can waste up to 60% of CPU time.

See what’s happening here? For the computer, it’s like when a random friendly sales team member randomly shows up at random times of thethe day to you, the random programmer, to say things like “sup man, hey yo IT pro what’s cookin man, hey, you must be doing some serious thinking and shit, but man, hey, just gotta check you out and ask what’s cookin’, man, so, hey, man, what’s cookin’?” and then look at you with big, brown eyes waiting for an intelligent answer.

This not only wastes aerial brownian motion, but can also suck up to 60% of your brain power (long term effects not counted). No wonder the poor, idle machine can’t get things done any more, because the CPU is clooged with the interrupt controller gone haywire incessantly screamin “SUP, MAN! YEAH!” It’s like Guantanamo for geeks. It’s not torture, bit it is enhanced communication.

The most obvious solution is to fire the sales team.

followed by

This works like a charm. The sales team is gone, the chatter stops, and the CPU is free to work again (not to mention the tremendous emotional satisfaction).

However, it has a big minus. I have found that the sales team, properly kept outside of the IT office (kind of like farmers put wire fence around trees in goat pens), actually gets things of its own done. More specifically, I realized my Indigo IOx Expresscard
was no longer working, which was the reason I bought my beloved Lenovo in the first place. I hate to admit it, but firing the sales team is actually kind of a bad idea. Uh… sup guys, wanna come back, like, uh, would you?

Luckily, a very smart fellow on the linux kernel bug forum found a better solution: Instead of firing the entire sales team, we only fire the team member that does the most interrupting. Sales can still be made, we can work, all is well. First we need to find the culprit:

This simply clobbers the interrupt that is problematic, while the express card still works great.

Oh, and I’m kidding about the sales team. I love you guys.

I need backup. The time I almost lost 80 hours of work tought me that. And I can’t trust myself to handle the encryption. The guy who lost his life savings in bitcoins tought me that. And me, at four in the morning on a work day, trying to get GPG keys right for duplicity.

I also need about 200GB off site for stuff that I will never use again but want to keep. Enter Glacier.

Glacier will let me keep those 200GB online for 2 bucks a month. Ridiculously cheap, no? That’s what I thought. The catch is, if you ever need to restore it, Amazon essentially does a random hash on your bandwidth usage and presents it to you as your bill. Seriously, that price structure is really, really hideous.

Why would Amazon do such a thing? I wonder. They don’t normally suck. They recently coaxed an on-line vendor into sending me three boxes of dry ice because I wanted to see what would happen if I ordered Yoghurt. No one has ever so disproportionately outperformanced my expectations.

To mitigate my distrust of Glacier I kept on looking around and eventually came across a comment somewhere by somebody who said he’s an ex Amazon employee. Apparently it’s all custom hardware, with custom processors (seriously? wow) and custom hard drives with really slow spin cycles to reduce wear and power consumption.

This didn’t make me like the price structure more, but it did help me believe that Glacier is more than just an elaborate scam. They can actually afford to be cheap. It the high retrieval fees reflect a trade-off with their hardware that actually exists, they are not just a way of preying on retrievers to subsidize those who don’t retrieve. They are Amazon’s safeguard against using their special custom hardware in a way that will break it.

What also helped is the realization that to have a safety net against an avalanche on your Glacier bill, you do a simple thing: Throttle bandwidth.

So I came up with the following strategy to sieze the opportunity that is Glacier:

* For big, important stuff I might need in a month, use it as secondary off site backup.
* For huge stuff I might need in 10 years, use it as the only backup. Keep the external hard drives but don’t worry if they rot.
* Make a bet that before I might need to restore the full monty, someone, somewhere will create a Glacier client that will just let me set a budget and a timeframe. As a last resort, set up a RaspberryPi and a USB drive and keep it downloading for a couple of months. An Amazon Export-By-Mail might do the trick as well.

A little note on encryption:

The crypto community will never stop badmouthing Amazon S3 server side encryption, and I understand why. It’s mathematicians who don’t care how big a number is as long as it’s not infinite.

Not screwing up encryption just takes too much time when it comes to backups. And with Amazon’s encryption, they store the keys in a different place than the files. This means that a bored employee will not be tempted to thumb through my vacation pictures, which is good enough for me. The manager who has the keys will probably be too clueless to decrypt the data.

PHP 5.4 is so nifty. And traits are mad cool.

And you might be in the same situation as me: Big product with a huge amount of customer benefit built on Symfony 1.4 with years yet to maintain it and expand on it.

So you might want to start using traits, just like me. The first thing to do was, of course, to just do it:

To which the feedback was very clear:

Fatal error: Trait ‘ImageUpload’ not found in /path/to/project/lib/MyClass.php

Symfony was not autoloading traits.

Which is good. Symfony autoloads selectively, and you might not want to autoload every stray php file a stray programmer might have forgotten inside lib. (kidding)

I found by poking around in the Symfony 1.4 sources that the method actually selecting which files get autoloaded and cached is sfAutoloadConfigHandler::parseFile. It contains a regex that filters files to contain a line that starts with nothing or whitespace, followed by the words final or abstract, and then class or interface.

So a quick and dirty way to get traits autoloaded is to include a commented out mock class or interface definition in the trait file.

Of course this is not pretty, but it will do if you need to autoload a couple of traits now and worry about clean code later.

To clean things up you can customize sfAutoloadConfigHandler by customizing configuration files.

First, create a custom configuration handler:

Then start using it by creating or adding to your existing config/config_handlers.yml

Then clear the cache, and your trait should be autoloaded and cached by symfony, in just the same way interfaces and classes are.

To stay on the really really safe side, it is a good idea to monitor possible Symfony 1.4 maintenance releases for changes to lib/config/sfAutoloadConfigHandler before upgrading. It’s unlikely to happen, but it might.

And this is how to get using a really powerful PHP 5.4 feature with your existing Symfony 1.4 project. Yay!