Search and replace on a large file on OSX using Sed

This post was written 2 months ago.
Thu, 06 Mar 2014
My code editor of choice, Sublime text 2, crashes when you try to do a search and replace on a large file. This makes it difficult to do a search and replace on a large sql file, as I sometimes need to do when moving a wordpress site from one domain to another

The command-line tool sed can be used instead like this:-

sed -i '.backup' 's/olddomain/newdomain/g' wordpress-backup.sql
This makes a backup of the original file, then updates the original
Tags: wordpress / osx /

Rasberry Pi as low power office server

This post was written 3 months ago.
Mon, 20 Jan 2014
raspberry pi running as low power office server
Part of a system that I look after for a small distribution company includes a linux server based in their offices. The server doesn't do much - it runs a few cron jobs and a mySQL database, acting as a monitoring system on the internal network and as a middle-man receiving data extracted from Sage Line 50 database on a windows server and posting off data updates to a web-based server. It is however a vital part of the system, and needs to be running 24/7. For years this has been handled by an assortment of dusty old desktop PC's, the last of which was starting to show signs of needing to be retired. The next old PC in line turned out to have a faulty network port, so I though it might be time to buy some kit specifically for this job. It occured to me that due to the low power requirements, a Rasberry Pi might be ideal.

Being fairly new to the Rasberry Pi, I ordered a starter kit including a Model B Pi, a 4GB SD card with "noobs" installed on it, a case and a power supply. Connected up to the TV at home via an HDMI cable, it booted into noobs, from where I installed raspbian linux. Once raspbian was installed, I found that I could install everything I needed (mysql, PHP5, apache) using apt much as I would on a regular debian-based linux box. One thing that I found I couldn't do (from googling rather than actually trying) was install dropbox, however this wasn't essential.

After installing it in the office and monitoring it for a few days, it seems to be coping extremely well. There is one particular php/mysql task that sends the cpu temporarily very high, but I think this can be tamed with a bit of code optimisation. I also wish I had started with a bigger SD card, as there is only just enough "disk" space when everything is installed.

One advantage of this low-power system is that I have it running from a USB cable plugged into another server rather then requiring another port on the office UPS. I've considered having it run from a rechargeable USB power pack, kind of like a mini dedicated UPS. I'll be interested to see how it stands up to this kind of use. Due to the low cost of this system, the plan is to have a spare Rasberry Pi or two and some spare cloned SD cards on standby.

One other thing to note is that to plug the Pi into a monitor you may need an HDMI to DVI cable, unless you have a monitor with an HDMI port, or some HDMI or RCA to VGA solution (that will be another blog post). I only needed to plug it into a monitor to congure the static IP address - after that I ssh'ed into it from another machine.

So why didn't I just use a virtual machine? A couple of reasons: firstly, one of the tasks for this machine is to monitor the windows server from which the sage line 50 data is sent. Secondly, the aforementioned windows server is the only machine on the network running 24/7 and already struggles a fair amount, and I didn't want to hog any more RAM or cpu with a virtual machine.

Update 2014-01-22: The Pi has been up and running for nearly a week with no issues, but i've been reading about SD card wear that can occur if the disc is being written to frequently, so I plan to move the MySql and log file storage to an external drive, otherwise I suspect the card might "wear out" fairly quickly.
Tags: raspberry-pi / linux /

Managing Google App Engine SDK versions on OSX with virtualenv

This post was written 5 months ago.
Wed, 13 Nov 2013
When I first started developing Django apps on Google App Engine (GAE), I downloaded the google app engine launcher. Installing this will install the latest version of the SDK, and will (if you let it) update your path to point to it. This is fine if you are always using the latest version and don't need to use an older version for some reason. If that is the case you will need to uninstall GAE launcher, find an old installer for the version you want and reinstall.

A more manageable way is to not use the GAE launcher, but to download the SDK manually, then set up a virtualenv to use a particular SDK. I also use virtualenvwrapper to help me switch easily between environments.

This is the pertinent bit - in the folder where your virtual envs are stored, add the path to the app engine sdk in bin/postactivate. Postactivate gets run when you activate the virualenv through virtualenvwrapper. E.g for something using version 1.8.8 you might have something like:-

#!/bin/bash
# This hook is run after this virtualenv is activated.
export PATH=/Users/rickhurst/gae-sdk/1.8.8/google_appengine:$PATH
Tags: osx / appengine / virtualenv /

Everyday Designer Article: Freelance to permanent and back again

This post was written 6 months ago.
Mon, 21 Oct 2013
screen grab of everyday designer article by Rick Hurst
Earlier this month I wrote an article for Everyday Designer talking about my experiences about my experience of working as both freelance and permanent web developer.
Tags: freelance / articles /

Django Talk interview

This post was written 6 months ago.
Fri, 18 Oct 2013
screengrab of Rick Hurst django talk interview
I was interviewed for django talk earlier this month, and here it is.
Tags: django / freelance /

Catching up - Summer 2013

This post was written 7 months ago.
Mon, 07 Oct 2013
Rocky the T25 being towed
Ok after a productive start to the blog this year, i've failed to post since the start of the summer. I'm going to blame it mostly on my camper van, both literally spending time working on it and camping in it with my family, and also generally all my blogging effort has gone into my "camper van things" blog. I've had a fantastic summer, worked less than I probably should have, but now getting back to work (and saving up for some more summer adventures next year!).

So to catch up - i'm now freelancing more or less full-time back at Potato, working on some interesting stuff, notably cutting my teeth with AngularJs framework, which i've enjoyed so far. My initial observations are that it is significantly different to Backbone/Marionette that there is no confusion over what technology i'm using, it allows you to be productive very quickly and that it litters your markup with non-standard attributes. The latter may upset purists, but otherwise it gets the thumbs up from me. If you're interested in learning AngularJs, a good place to start would be the tutorial.

Braindump June 2013

This post was written 10 months ago.
Mon, 01 Jul 2013
Where exactly did June 2013 go?

Braindump May 2013

This post was written 11 months ago.
Fri, 31 May 2013
Not for the first time i've found myself stepping back from Twitter most of this month, and when I got drawn in, I got very frustrated with scatter-gun discussions - found myself trying to explain stuff I just can't get over in a few characters, resulting in multiple replies especially when there are several people in the discussion, so even less room for content. As someone once said - having a discussion on twitter is like trying to have a conversation through a car window as you repeatedly fly past them.

One such discussion was about recommended Content Management Systems (CMS) - I was trying to explain what my holy grail would be - a dynamic CMS (as opposed to a static site generator such as Jekll), which allows editing of content as text or markdown files or via the web, where content can be synced and versioned using GIT, but also has a "throwaway" database for search e.g. it indexes content in the background but no need to keep restoring it when content is added to live but not dev/ vice versa. I struggled to get that out in a reasonable paragraph and i'm not really started yet!

Another area to deal with is comments - comments should really be counted as valuable content on a blog, so this content should also be stored as static files under version control (to allow it to be synced with other copies of the site, rather than the comments themselves needing versioning). The problem is that 90% of any comment system is anti-spam measures and moderation. Having spam comments committed to my GIT repository is downright dirty!

In other news, I've finished my current stint as a JavaScript contractor, and am going to be working full-time for the next month or so on the next phase of my company Olivewood Technology, which will be evolving this year.

Braindump April 2013

This post was written 12 months ago.
Tue, 30 Apr 2013
One of the oldest production web applications i've ever built is still happily running mission-critical operations for a small distribution company. I'll be the first to admit that it is a bit of a mess of different "eras" of code - some of it is even still running classic ASP, which is being slowly (i.e. over a period of years) but surely replaced with PHP5 code. One thing i'm sure about though is that to rebuild the system entirely using something like Django would take months, if not years. So it is is destined to stay as PHP, and therefore as long as it is running and i'm involved with it, i'll still be writing PHP. I'm fine with this, PHP runs half the internet, was written specifically for the internet, is easily learned and is improving as a language all the time. PHP is a fully Object Orientated language these days, but judging from a recent thread on the underscore mailing list, the perception of PHP remains as warped as ever in the eyes of "proper" programmers!

I've managed to sort a couple of things this year that have been missing from my life - firstly, i've been doing a bit of mobile working from my camper van office, and secondly i've joined a band, after a hiatus of ten years! My last band (Braxton Hicks) was a 9-piece Acid Jazz/ Funk/ Soul band, this new band (name still undecided) is a 3-piece surf-rock trio. We're still getting a set together, but will be looking for gigs soon!

Bristol surf rock trio the savages

Braindump March 2013

This post was written 1 year ago.
Sun, 31 Mar 2013
There seems to be a trend for free services shutting down recently - following the recent-ish announcements that Posterous (a kind of blog service with strong emphasis on posting by email) is closing down, and this month Google announced they are closing Reader. It's not surprising that a free service would shut down - they cost money to run, and as businesses start to look at ways of cutting costs, it makes sense to sacrifice free services.

I was surprised about Google Reader though - apparently the user base isn't as big as I presumed it was, and they decided it was too niche to bother with. I'm also surprised they didn't just roll it into google plus - surely little effort to gain thousands of much needed G+ users? Meanwhile i've moved over to feedly.

Currently most of my stuff is self-hosted, with the exception of comments hosted by Disqus. I really don't want the hassle of running my own comment system, due to spam, but i'm starting to think about moving to a self-hosted system, maybe something like a self-hosted version of Disqus, that I can easily drop into different blogs? Whatever I use, it needs to have all the usual anti-spam stuff - comment spam is the bane of any aspiring blog software developers life!

I also paid up for a years membership to app.net - same as twitter, I am @rickhurst. It hasn't become part of my daily social/workflow yet, as not enough of my regular twitter contacts are using it, but I wanted to put my money where my mouth is when I say i'd rather pay for a service than rely on annoying advertising.

I gave a talk on Backbone Marionette at the Southville JS meetup. It was a practical code based demo, I think it went OK. Either way it was a good turnout and nice to have it hosted in a pub back room, so we could continue chatting at the bar afterwards. The code of the (fairly pointless) demo app I put together in blue peter style is online here.

I also gave an ignite talk on remote working at the Anywhere Working Bristol event. That seemed to go down well and it was nice talking about non code-related stuff for once!

I was also pleased when a colleague pointed me to Poweramp for Android - I had been complaining about how most Android music players don't support gapless playback, which is a real party-pooper when it comes to mix albums such as my favourite mix album of all time Journeys by dj 70 minutes of madness.

I've now finished a full-time freelance Javascript contract at ISM Fantasy Games, but will be continuing with them on a 3 day week basis, freeing up some time for my new business venture(s). More about that when the plans are more refined!