Category: Computing

Posts about computing and computing related matters

Posting to WordPress MU by email

Warning: This post contains material of a technical nature, if you start reading and feel dizzy or nauseous then stop reading immediately and go and lie down.

NOTE: The code and some of the details in this article have been superseded by a later post

There are a lot of clients for the PC/Mac/Linux which allow you to post to your WordPress blog without having to log into the back end, and some might question the need to be able to send posts into your blog via email but there are several situations where you can’t use a fat client or you don’t want to use the web back end for various reasons

WordPress MU (the multi-user/multi-blog version of WordPress) supports a basic post by email feature but it involves each user setting up their account to go out to an external email address and picking up the email. This seems rather “backwards” as it means your WPMU server is having to go out to find your new posts, and if you’ve got a lot of blogs then that’s potentially a lot of outgoing POP3 requests and it means that your have got the email account and passwords for your users, which some might see as not being a very good idea. WPMU suggest using a specific account to support posting by email and keeping the account “secret”.

Surely it makes sense for the posts to be delivered to the WPMU server and processed there. This has several advantages when you think about it.

So I’ve written a new version which uses a local virtual domain and a single pop3 box.

So the user emails theirwpmuloginid@some.domain.here which ends up in a pop3 mail box on a server somewhere (preferably the same server as WPMU). Every email sent to some.domain.here arrives into one single pop3 mail box rather than being in different ones. This “catch all” account means that you don’t have to do anything clever like creating email accounts for each user as they sign up.

My process then opens that pop3 account and reads each email in turn.

It will only post if the sender email ID matches the username it is associated with (from the user table). So you have to send email from the email address you signed up with, or the email address you’ve set it to in your account settings and you have to send it to the username you use to login to the site as. This is only very basic security and I will be adding more but for the first pass of the code I felt that this was good enough to prove the principle.

So apart from allowing me to post to my blog via email what else does this “extension” provide? Well users can specify some “options” in the email which affect how their post is handled:

There are currently 4 options that they can provide:

wppbm-tags: comma separated list of tags for this post
wppbm-status: publish [default] | draft | pending
wppbm-type: post [default] | page
wppbm-comments: open [default] | closed

All of these options are optional, you do not have to provide them and if you don’t provide them then the defaults indicated will apply. WPMU developers who have played with the wp_insert_post function will probably recognise them.

Posts accepted by email will be assigned to the default category for new posts.

It currently accepts basic html marked up (bold, italic, underline) messages (either manually coded or from the rich text editor in Google mail). Yahoo marks its html messages up differently so its not working if you try to use their Rich Text Editor, and “advanced” html mark up (like font sizes etc.) don’t work from Googlemail either. Again this will be addressed in a future release.

Things that need adding are:

  • A PID field (so that people can’t just “fake” the sender email address).
  • Ability to support posting to different blogs (rather than just primary).
  • Ability to support posting from more than one email account.

So if people want to try it, and leave feedback then this is how you implement it. If you can’t edit your Postfix configuration (or don’t know how to) then I’m sorry but I can’t help you. I don’t know if other MTAs support virtual domains but if they do and you get it working then we can add those instructions.

I’ve only tested this on Linux so you’ve been warned!!!

In these instructions I’m using posts.my-dummy.domain to represent the domain we’ll be using to send emails to.

Step 1 : DNS Records
Configure your DNS records to support posts.my-dummy.domain. You need to be able to send email to this domain.

Step 2 : Create a REAL email account on your server.
However you do it you need a local email account on the server that you’ll be directing email to. In this example I’m going to call this user wpmuposts

Step 3 : Postfix Configuration
Edit your postfix main.cf file and add the following two lines:

virtual_alias_domains = posts.my-dummy.domain
virtual_alias_maps=hash:/etc/postfix/virtual

NOTE: The virtual_alias_domain MUST NOT be listed in the mydestinations configuration parameter.

Create/edit the /etc/postfix/virtual file and add the following line :

@posts.my-dummy.domain wpmuposts

Save the file, and then you need to create a hash for it using the postmap command. I do this by going to the postfix directory (/etc/postfix on my server) and typing:

postmap virtual

You may then wish to check that everything works by waiting for your DNS to propogate and then going to an external email client and sending an email to anything-here@my-dummy.domain.

You should end up with an email sitting in the POP3 email box wpmuposts. If not then you need to review your settings. There is no point in going beyond this point unless you’ve got the “catch all” email working.

OK so email is now working. That really was the hard part. The rest is pretty easy, and if you’re doing this then going in and editing a php file isn’t the sort of thing that scares you right?

Step 4 : Getting and configuring the extension

Grab a copy of post_by_mail.php and save it to your computer.

Open the file in your favourite text editor and look for the following :

// USER NEEDS TO SET THESE.
// Postfix will need configuring to support virtual domains with a wildcard to deliver to the account given below.
define ("POP3HOST","your host here");
define ("POP3PORT","110");
define ("POP3USER","special pop3 user account here");
define ("POP3PASS","special pop3 user password here");
///

and edit them to match the POP3 settings you need to access your special POP3 email account: So for example:

// USER NEEDS TO SET THESE.
// Postfix will need configuring to support virtual domains with a wildcard to deliver to the account given below.
define ("POP3HOST","localhost");
define ("POP3PORT","110");
define ("POP3USER","wpmuposts");
define ("POP3PASS","notarealpassword");
///

Save the file and then upload it to the ROOT of your WPMU installation. DON’T FORGET TO REMOVE the .TXT extension!

That’s basically it, the extension is there. If you want to give it a different name, for security reasons, then feel free to.

To test it, send an email (and I’d suggest setting the wppbm-status to draft) to yourwpmulogon@posts.my-dummy.domain. Then once its arrived call:

http://my-wp.domain.here/posts_by_mail.php

You should see some diagnostic message on the screen and hopefully, fingers crossed, you’ll end up with a new post in your blog.

So all you then need to do is set up a cron job. At the moment you can’t use the PHP CLI to run this code but you can set up a cron job to to a wget on the URL. As this isn’t a silent process I don’t recommend putting a call to it into your themes, its much better to keep it controlled by cron (and that way you can route the output to a log file to keep an eye on it).

Debugging
If you think that there are problems and you want to keep checking the same email as you add debug code to the extension then you’ll want to stop it deleting emails / posting messages.

To stop it posting into your blog look for the following line (line 241):

$post_ID = wp_insert_post($post_data);

and put a # in front of it.

To stop it deleting messages on completion look for the following code (line 256 – 262):

if(!$pop3->delete($i)) {
echo ' <p>' . sprintf(__('Oops: %s'), esc_html($pop3->ERROR)) . ' </p>';
$pop3->reset();
exit;
} else {
echo ' <p>' . sprintf(__('Mission complete. Message <strong>%s</strong> deleted.'), $i) . ' </p >';
}

and comment it out.

So there you go. Please remember that is a work in progress and support for complex HTML isn’t there and you install it at your own risk. If you’ve got any suggestions on improvements, or you’ve found a bug then please let me know.

Getting to grips with Latitude


Warning: This post contains material of a technical and mathematical nature, if you start reading and feel dizzy or nauseous then stop reading immediately and go and lie down.

Google Latitude is an interesting little app which allows you to advertise your location. On the Android G1 the GoogleMaps application comes with the ability to update your Latitude location directly from the phone.

The usual way of displaying your Latitude location is using the iframe that Google provide which looks like this:

Part of the integration work that I’ve done to link the blogs here to my brother’s CanalPlan website (which is based very loosely on work I did for my ‘O’ Level computer studies way back in 1981/82 – and yes we did actually have microcomputers back then) involved using his list of locations to allow you to insert links easily into the blog post . I then extended that to allow you to easily embed Google maps of those locations into your posts. For example Wigan Pier  :


So the next logical step was to link Latitude locations back to CanalPlan locations.

first of all I looked at scraping the contents of the iframe but its a complete mess, and apparently against the Google TOS, and I was struggling of finding a neat way of getting data from Latitude. Then I found that Google provide two “feeds” for third party applications (i.e. its acceptable under the TOS to take data). The support extracts in JSON and KMZ formats. I choose the JSON one (http://www.google.com/latitude/apps/badge/api?user=-2983480754731033286&type=json)  because I’m already parsing JSON data from my brother’s site so I could just re-use some of the code. So that’s brilliant, I now have my Latitude and Longitude from Google Latitude and I have a list of Canalplan places with latitudes and longitudes so in the words of Aleksandr Orlov, Simples, just match the two and the problem is solved.

Actually its far from simples because the locations inside CanalPlan are for very specific points and the chance of ever being on exactly the same co-ordinates as the CanalPlan location are pretty damned slim, especially given that a lot of the locations in CanalPlan are derived from other data sources. But it would be really nice to able to show the nearest Canalplan location to your current Latitude location wouldn’t it? Especially if you were on the boat and moving.

There is a solution: Geospatial functions, and I think I just head a couple of heads hitting tables.

I was lucky that I’m running the blogs on MySQL 5.x because it supports some basic Geospatial functions. These are functions that allow you to perform calculations on places or rather sets of places, in my case that’s the latitude and longitude of two places. I had to create a new column for my places table (called lat_lng_point) which is of a special datatype called point, and then I had to populate that column with data calculated from the latitude and longitude for each place using the MySQL GeomFromTextfunction which creates a very odd looking data item.

It might look very odd but its quite powerful because you can do some very interesting things with it. If you take two “points” created using GeomFromText you can find the “distance” between them using the GLength function.

So using the following bit of code:

$findit="GeomFromText('Point(".($lat) ." ".($lng).")')";
$sql="SELECT place_name,lat,`long`,GLength(LineStringFromWKB(LineString(AsBinary(lat_lng_point), AsBinary('".$findit."')))) AS distance FROM canalplan_codes ORDER BY distance ASC LIMIT 1";
$res = mysql_query($sql);

We can take the latitude ($lat) and longitude ($lng) from the JSON feed and find the nearest CanalPlan location that matches it.

At the time of writing this post the code (with a few extra bits to make it look nice) produces:

Finding places closest to My current Latitude location at Cheltenham, Gloucestershire, UK (51.9031645,-2.0471622)

Avon Lock (51.9962,-2.15681) is 0.14382 from Cheltenham, Gloucestershire, UK

Now no doubt you’re scratching your head trying to work out what that distance is. Well its a distance expressed in degrees which is pretty meaningless to people like us.

So that is how things stand now, and that’s why the little Latitude Widget on the right hand bar has a little bit of text underneath it saying what the nearest Canalplan location is.

What comes next?

Well possibly implementing a “real” distance. As I said the distance given is in degrees so we can either trust that its right or we can recalculate from scratch. The calculation for this is quite simple (!) :

gcdr=acos(sin(lat1)*sin(lat2)+cos(lat1)*cos(lat2)*cos(lon1-lon2))

This gives the “Great Circle Distance” between the two points (lat1/lon1, lat2/lon2) in degrees. You then convert this to nautical miles by multiplying it by 60, and then finally multiplying that by 1852 you get the distance in metres which I can then convert into one of several “nicer” formats. So it could say:

Avon Lock is a distance of 9 miles, 7¼ flg from Cheltenham, Gloucestershire, UK

The other, much easier, thing to do is to create a new widget that replaces the Google Latitude iframe with a mini Googlemap with your nearest CanalPlan location on it (and saying where you actually are under it).

Well I think everyone’s brains are probably hurting now, so I think I’ll go for a walk.

Class Dismissed!

Internet on the Boat

Kathy and I spent the weekend on the boat on the moorings. We couldn’t go any where because the canal is closed due to a breach on one of the embankments and so they’ve closed all the locks to stop boats moving and hopefully to keep the water levels up until they can get some pumps in to pump water round the section they have had to drain.

My G1 is still away for repair and so I’ve been using Kathy’s old LG Viewty which has some very odd quirks to it and I’m glad its not my normal phone.

There is one good thing with the Viewty in that Linux just recognises it when I plug it in and I can use its 3G modem without installing any software (unlike Windows where you have to install the unbelievably crap LG communications suite.)

When I tried it at home it had crawled because we live in a 3G dead spot, but out on the boat moorings it really flew.

3G speedtest on the boat.
3G speedtest on the boat.

Yes, that really is a 665Kbps download speed. I think if I try it at home I get something like 86Kbits.

So I configured my laptop to share its internet connection and we actually were both surfing the net at the same time. The only down side is that the 3G stack seems to choke if you ask it to do more than a couple of things at once.

TasKiller

I’ve had an Adroid G1 on T-Mobile for quite a bit now and I quite like it, its got a lot of good features but one huge problem with it is the damned battery life. Like its terrible.

I found I can make it last longer by turning off 3G unless I know I’m in a good 3G signal area. My part of Cheltenham is lousy and sitting on the table at home it will often lose data connectivity completely.

But the other day I stumbled upon TasKiller which is a totally free app in the Google Market Place.

So what does it it do?

Well it kills apps, pure and simple. When you “quit” an application on the G1 you’re not really quitting, its just swapping things round. Now some of these apps don’t really cause problems but others, like the built in email app, seem to hog resources.

After installation you simply start it up and it shows “idle” apps in white and active apps in yellow. Short clicking on an app kills it, if you long click on it then you can either switch to it, or you can add it to an ignore list.

The ignore list is very clever because one of the features of TasKiller is you can kill all running apps except those that are on your ignore list.

So that’s a quick overview of what it does. It doesn’t sound like a lot but it really makes that battery life last longer.

If you’ve got a G1 then get this app, seriously.

Keeping things up to date

It’s a never ending task really.

Most of the core server components update automatically, as long as I remember to kick off an update process once in a while, its everything else that you need to keep a eye on.

The server has a WPMU installation on it which has to be upgraded, as do all its plugins, and then there is the WP install and its plugins, then there’s the photo galleries which use Coppermine which has to be updated.

Pubnight doesn’t escape either as the Wiki software has to be updated as well.

Even when I upgrade the server it can trip you up because upgrades on the Apache server / php packages usually break eAccelerator which then has to be recompiled.

Still its keeping me busy

Hackety Hack

So what have I been up to?

I’ve been working on code for Pubnight and really just adding a few features to make things easier for people editing and adding information, basically Richard who has been hard at work on updating the site.

Part of the enhancement was easy, actually two bits were. I added two new custom tags to the Pubnight code which allow quick linking to beers and breweries. The other easy bit was adding a Rich Text Editor using the FCKEditor extension.

The hard part was adding extra functionality to the FCKEditor. I added a form which allows the easy selection and insertion of PNC Pub Digraphs and beer and brewery links. So now rather than having to remember that The Red Lion Inn, Market Drayton has a Digraph of RM you simply start typing the name of the pub and then pick it from a list. It uses the same code base as the canalplan codes function on here which is based on the same code Nick uses on CanalPlan. So that’s a good re-use of code!.

Pub Night

Pub Night is a website that I set up for a group of friends who go out drinking once a week in Cheltenham. We’ve been doing it for years and a few years ago now Nick and I came up with the idea of building a website for it. I went for a Wiki based site but because we wanted “standard” information for each pub I built a rather large extension for it which basically allows us to use Data pages to hold the information on the pub and then for each pub we have a simple page which contains some tags which allow us to display the information in a controlled format.

Pub Night is primarily based in Cheltenham but has details on some pubs outside of the town, primarily in Gloucester,  London,  Upton upon Severn and Market Drayton.

There is also a sub group of Pub Night called Out Of Town Pub Night Group (or OOTPNG for short) which once a month goes to a good pub within a reasonable driving distance of town. The venues and drivers and approximate dates had been decided and I went to load them into the system so I could put up a page with the 2009 locations. That is where the problems began. The code that drives pub night talks directly to the database behind the wiki which means that information on pubs normally comes out in the order the pubs were added into the database. To support the new page I added two new fields to the data set – nextvisit and driver. But I wanted to sort the information on the nextdate. So I’ve had to go in a recode how the “where” part of the extension works so that rather than just trawling through the pubs in turn and if they match the criteria displaying the information I now go through the pubs once and grab a “sort” field from the data (if specified) and then I sort the list of pubs on that and then process each one in turn to get the information. Sounds easy doesn’t it?

Well it is when you are dealing with text fields because you can simply build the array with the Digraph and the “sort” field and then use the php functions asort and arsort. Date fields are another matter entirely. Pub Night stores its dates in dd/mm/yyyy format which do not sort. PHP has a strtotime function which converts “human” date formats into unix timestamps. Those we can sort easily so problem solved. Afraid not because strtotime doesn’t like dd/mm/yyyy. So I had to us the php mktime function to parse the PN date format into something I could then pass into strtotime to get the timestamp and now its all working.

Now all I have to do is fix the problem with null dates which upset it (silently, they just spew errors into the error log!)

Things to do when you are unemployed

Well I’m still sort of offically employed : my last pay cheque arrives this Friday but I’m laid off.

So I’ve been using an old Toshiba Satellite SA30 laptop running Ubuntu and its OK but its only a 640*480 display and it was getting a bit unreliable. You know when the disk drive starting making twanging noises that its not a good sign.

So I went and bought a new Toshiba Satellite : a L300D-13S. Very nice, 3GB RAM 150GB HD, widescreen, AMD Turion X2.

All well and good but it comes with Vista Home Premium. What a pile of crap.

How can something like that take so long to boot up for Gods Sake. It takes forever to get to the password screen. It drove me mad, and dont even try to work out why applying system patches is rather like watching paint dry underwater.

So I decided to dual boot it with Ubuntu which at least uses the 64bit extensions rather than the 32bit Vista.

So I got that working after fighting with the partitioner and everything was fine. Until I booted back into Vista when the Wireless card didn’t work. Couldn’t get it working so I guessed I’d somehow screwed things up. Rebuilt it using the recovery DVDs which I’d made. Wireless came back for a bit then stopped again.

So I go round and round in circles. Ubuntu works fine on the Wireless. Vista works once then craps out on me each time.

So I dig round and what do I find. Its nothing I’ve done wrong. Its Vista being a total pile of shite.

I used WEP encryption with a 128 bit key on the home wireless. OK I know its crap but its better than nothing and the old 11b PCMCIA card in the SA30 didn’t like WPA.

Of course some moron at Microsoft screwed the WEP encryption on Vista and it looks like it just doesn’t work with 128 bit keys and truncates them at 64 bits. I did seem some advice that says “Upgrade to a Vista compatible wireless router”. I’m sorry? I end up buying a machine with an O/S that I have no choice over and now I’m told to upgrade my router to something that works with Vista because someone somewhere didn’t code something properly.

So I switch everything over to WPA and accept that I’ll have to use a cable to connect the old SA30 to get stuff off it.

Everything is fine until I really push the new laptop which then gets DNS failures and the only fix seems to be to restart the Wireless Access Point. So I drop back to WEP and it seems to be OK but of course Vista wont play. Then I manage to get the same problems under WEP. So I do some digging round and find that my new laptop is running IPV6 which seems to cause problems with the router.

So I turn off IPV6 on Ubuntu and its OK now so tomorrow I’ll put everything back on WPA and see if it goes wrong again.

OK so the Ubuntu stuff is not good – quite why it comes with IPV6 turned on by default I do not know but I’m sure it will be fixed soon. The router is old and I guess it can be excused not working very well. What is totally wrong is the fact that Microsoft can’t cope with WEP using 128 bit keys on Vista. OK I know WEP is broken  but that is NOT the point. WEP is a standard and once again Microsoft have decided that that they know better.

Tweaking a Calendar for WPMU.

NOTE: Kieran has made a newer version of his plugin which works with WordPress MU and integrates by using an embedded Calendar tag to display the calendar.

So basically I’m abandoning this plug in and suggest you get Kieran’s new version instead. That is what I’ve put on this site.

This is basically Kieran O’Shea’s WP Calendar plugin for WordPress tweaked to work with WPMU with a couple of additional features.

1) Calendar table created for each user
2) The Calendar is wrapped to work in multiple templates without needing a file in each template. See the section below for more on this
3) Input and Display date formats on the Admin screen can be changed by the administrator.

Installation.

1) Drop the files from the archive into the matching folders in your WPMU installation
2) Edit the edit-calendar file and change the $eddateformat and $eddatesep variables to match your date format (by default it they are set to d/m/Y and / as I’m in the UK).
3) Enable the Plugin
4) Edit your .htaccess file and add the lines in the add.htaccess file towards the top
5) If you want to use my template wrapping then you need to use the wp-calendar.php file that I have provided along with the two tag files (edited to support your templates). If you dont want to use it then you need to install the calendar file manually into each template (but you dont need the tag files)

Template Wrapping.

I use Template Wrapping a lot – it allows me to easily drop cross blog functionality into the blogs without having to create a new page template in each blog. For example the calendar page only exists at the top level – there are NO files in any of the templates for the calendar, the structure of the page round the calendar is created using the two tag files and a couple of calls in the file containing the call to the calendar code

To enable template wrapping you need to do the following:

1) Drop the wp_opentags.php and wp_closetags.php files into the top level of your WPMU installation
2) Create a new file from the empty_page.php file
3) The new page can now be accessed from each blog.

You will find that you need to edit the wp_opentags and wp_closetags files to work with your templates. This can take a while due to the way different people build their templates and no hard and fast rules on opening and closing DIVs in the header,page and footer.

A Zip file of the various files including my wrapper files is available for download

If you have any comments, suggestions etc then please post them as comments.

You can see the calendar in action here