Danny Tuppeny

Did Microsoft Just Kill Windows Phone 7? (Part 2, Answer: No?)

2010-08-18T20:26:00Z

Yesterday I posted some info that suggested a two-tier Xbox LIVE marketplace on Windows Phone 7, and that I thought it was a massive mistake on Microsoft's part. Today I had a lengthy chat with Paul Foster, a Microsoft Developer Evangelist, about that post. He wanted to give some more details about how things worked to avoid me having to draw my own conclusions.

We went into Xbox LIVE publishing in great detail, and I decided to get Paul to confirm some of the things we discussed by email so I could post them here, for those that had read my previous post and were worried. Paul was unable to answer all of the questions (due to various things being hush-hush at this time), but said he'll answer them when he can. Below I've included his answers, along with the outstanding questions we'll hopefully have answers for in the future!

Q: Will Microsoft engage directly with small companies/individuals that have great games, ready to publish (to become part of the managed portfolio)?

A: Microsoft's XBL team and importantly a range of 3rd party XBL publishers are looking out for new opportunities, in fact at GDC 2010 and Mix10 the wpgames@microsoft.com alias was advertised asking for indie games devs to submit their game concepts they are building. The important point here is that if you want your game to be a big success you have to do more than just post it into an app store. You need to think about it as a manuscript for a book and take it around various publishers to get opinion and seek publishing opportunities. Publishers have the skill and capability to market your game and find bigger publishing opportunities – such as XBL. Publishing by default into an app store isn't enough for success when big numbers of people are doing just that – you need to do more than just develop your game, you need to drive the commercial publishing of it and the ecosystem of XBL provides broad opportunities for publishing. If after doing all that, the best opportunity for you is to publish straight into the marketplace then work at advertising your game. Seek magazine or blog reviewers, publish videos of it and ensure you have a cool web landing page for it. None of this stuff costs much and can raise the profile of your game.

Q: How are managed/unmanaged portfolio games separated in the store (visually/category-wise)?

A: All apps/games are in the single app store, filed by various categories. There are various merchandising locations in the marketplace and in the Games hub – these will be driven by various metrics from the store as well as for apps/games of planned marketing activities.

The following questions are unanswered as yet, but it seems like the answers will be positive for indies!

Q: Will there be separate "top sellers" / "what's new"-type lists for managed/unmanaged portfolio games, or combined lists?

Q: Are managed/unmanaged games separated in search results (if I search by name)?

Q: Will there by managed/unmanaged portfolios of apps too, or games only?

Did Microsoft Just Kill Windows Phone 7?

2010-08-17T21:00:00Z

Update: Today I had a lengthy chat with Paul Foster, a Microsoft Developer Evangelist, about this post. I posted an update here.

Ever since I had my first iPaq with the .NET Compact Framework, I've wanted to code for a Microsoft mobile device. Coding in C# and Visual Studio is by far the nicest coding experience I've come across in the 15 or so years I've been programming. Unfortunately, up until now, Microsoft's mobile operating systems have all been absolutely rubbish.

Windows Phone 7 is set to change all that. Everything I've seen suggests Microsoft have finally got their act together, and WP7 looks set to be a huge success. Finally I can have the Windows phone I've always wanted!

Over the past couple of months, I've been reading more about developing for Windows Phone 7, and in the last few weeks, I've been playing around with both Silverlight and XNA as part of this. The experience is great, and I've been really looking forward to working on a game for WP7. I've spent the last few months of my spare time working on an iPhone app, and it's so nice to be able to switch Xcode/Objective-C for Visual Studio and C#!

However, there may be a problem...

I was just pointed to this page by a fellow XNA coder. Although not entirely clear, my interpretation of this text surprises, and shocks me. (Emphasis mine).

To answer, it's important to understand that we have a managed portfolio for Xbox LIVE titles on Windows Phone 7. As we looked at how the mobile gaming industry was evolving, it was obvious that there's a significant problem with the "race to zero" – or having too many, mixed quality titles available on a marketplace. Too many offerings makes it very hard for any specific title to stand out (no matter how high-quality), and negatively impacts the overall business ecosystem by making it almost impossible for developers to earn a fair return on their work. Our goal with the Windows Phone 7 games that utilize Xbox LIVE services is to cultivate a portfolio of high-quality titles that are regularly released – this is very similar to the portfolio management we do for Xbox LIVE Arcade titles.

This sounds to me, an awful lot like the current Xbox LIVE Arcade situation. Indie titles (eg., those created by people that do not have commercial relationships with Microsoft) are relegated to a crappy sub-category of "Indie Games".

You may be wondering what the big deal is here - surely we want to keep all this "indie crap" generated by bedroom coders hidden away so we can see stuff output by the real companies, eh? Well, not so. Not only are there "real games companies" that develop this way (because they're just not big enough to have this kind of relationship with Microsoft), there are also a lot or really good quality games output by "bedroom coders".

If you look at the Apple/iTunes App Store, every application or game sits equal. A game I publish, sits alongside a game published by a large company like EA. There are "league tables" for ratings and sales, which allow the best games (whether they're from a large games publisher, or a single coder in his bedroom) to rise to the top. For all of the faults with Apple's App Store, this is how it should be. If an individual puts out some work that is better than EA, then it deserves to be above them. Consumers care games, not about the companies (or individuals) behind them.

I have no real idea, but I would estimate that at least 75% of all apps/games on the App Store are created by individuals or companies too small to have relationships with Microsoft. That is a huge portion of apps/games to relegate to "the shit section". One of the reasons the iPhone did so well was because of the third party software. If Microsoft screws over the huge majority of developers like this, it will absolutely affect the success of their platform. We're already lacking things like Achievements - it's already going to be hard to convince users they should buy our game instead of EAs's, so pushing us into the amateur category will just make things harder (And therefore, less worthwhile).

My interpretation of the quoted text may be incorrect. I really hope it is. It's also possible this affects only games, and not apps. Who knows? If games created by me are relegated on Windows Phone 7 like they are on the Xbox, then I'll simply put up with Xcode/Objective-C, and go back to coding for Apple's platform. And this means, despite my desite for a WP7 phone, I'd probably end up buying a new iPhone, and not a WP7 phone.

Until this little mystery is confirmed, one way or another, my WP7 game is on hold :-(

Giving up on Google Buzz (and why)

2010-08-16T18:35:00Z

First off, let me get this out of the way before you get the wrong impression of me...

I HATE TWITTER.

It's always down, it's full of spambots, and 140 characters is just too short. If I find myself going back to shorten words to fit like an SMS, then I'm afraid it's something wrong with Twitter. It can't be me because I don't know any big words.

So, with that out of the way, you know I'm not a Twitter Fanboi. Infact, you can probably conclude that I'm a Google Fanboi. I host my domains at Google, I use their email (Apps for Domain - all free!), I even wrote this blog in Python - a language I've never used before, because I love App Engine. I'm a Google Trusted tester, and have been beta testing new features in Buzz for a few months. I'd go and work for Google at the drop of a hat.

Hate Twitter, Love Google, but Ditching Buzz? WTF?

It might sound a bit crazy, but there are good reasons. And in the interests of being constructive, I'm going to list them here. Some of them might (will) contradict things said above, but tough. That's how it is.

Not enough people using it. It's catch 22, but all the people I want to follow do not use Buzz, so I have to use Twitter too. I don't need two services doing (mostly) the same thing.
Posts are too long. This might sound stupid given I said Twitter was too short, but the problem with Buzz is that it takes too long to catch up with things. I should be able to skip over a summary of posts, with a link to read more for the things that entice me. Twitter got the number of characters wrong, but the idea of restricting things right. I can skim over 24 hours of Twitter in 5 minutes. Not true of Buzz.
Google Buzz still can't sort properly. This one >really bugs me. Googlers suggested muting, but this is a pain in the ass, and will also break Google's clever algorithm that learns what I want to read. I want to read it - I just don't want comments sending it to the top!
There are no decent iPad clients. I love Twitterrific on my iPad. Buzz doesn't have any decent clients (seemingly on any platform). I don't like web-based apps, they just aren't written to work with the device, and they're slow. I totally get that Google want to be web-based and work on all platforms. That really is great - but they will definitely lose out a little against clients written specifically for a device.

Some of these reasons are linked - eg. a decent client would surely sort properly :-) However, one of the problems that a third party can't solve is that it's impossible to render a decent "summary" of a Buzz post. The web application shows the first few lines, but this rarely represents the content of the article.

I totally believe Google Buzz could easily become a better service than Twitter (Apples to Oranges, I know, but there isn't really need for both). Unfortunately it has some major issues that just make it hard for me to use. It's entirely possible that my use case is wrong, and that it's a fantastic service for other people. For me, however, it just doesn't fit. It's like a lame version of Google Reader, without the awesome third party iPad apps.

In the interests of getting feedback from people that might not visit to leave comments, I'll be following this post in Google Buzz too, though it's probably the only thing in Buzz I will be following!

Breeze - Xbox LIVE Indie Game

2010-08-15T17:53:00Z

Rob Hutchinson, a talented programmer I used to work with, from nullcity.com has just release a game called Breeze on the Xbox LIVE Indie Marketplace. It was built using Kitae, his 2D XNA Game Engine. I'm really impressed with how this game turned out, and I think it's going to do really well in the marketplace. I thought I'd plug it here, as I know many readers are XNA coders :-)

XNA Games Competition @ Work

2010-08-11T20:20:00Z

With the iPhone app I've been working on nearing completion, and Windows Phone 7 + XNA 4.0 in sight, I recently decided to get back into game creation. I've previously released a game on Xbox LIVE Indie Games, Jungle Blocks, but it wasn't very good, because I was somewhat excited about the idea of my game being live, and pushed the button before it was really finished.

So, here I am again. Take 2. This time, I want to finish something. A game to be proud of! It doesn't matter how long it takes to finish, I've already done the whole XBLIG thing, so I aim to take my time and do it right.

In order to keep me motivated, I decided to encourage my colleagues to get involved, by declaring a games programming competition. Until now, nobody else at work really knew XNA, so by getting others involved, there will be people to talk to and discuss ideas with. To encourage people to join in, even if they have other priorities, we set a 6 month deadline and set down minimal ground rules. To keep things fair, we decided:

Games must be legally publishable - e.g. no copyright material
No paid-for assets

We didn't define a platform/technology, to allow for iPhone, Android and other platforms, but as it turned out, all but one participant decided on XNA :-)

It would be kinda lame for someone outside our company to win the competition, so I'm afraid you guys are official excluded. However, if you're interested in learning something new, I encourage you guys to lay down a similar challenge at your workplaces. If you do, be sure to let me know - I'd be interested to see how other people get on with the same challenge :-)

I was toying with the idea of creating "Jungle Blocks 2" for the Windows Phone 7, since it wouldn't be hard to do, and if it was ready for WP7 launch, it could probably make a few quid before the marketplace fills up. However, because I have a (small, but real) advantage over my colleagues (having used XNA before) and because a few of them are ambitious enough to be doing 3D, I feel somewhat pressured into not being a wuss with a 2D game...

So, my plan is now to build a small 3D RPG for the Xbox, something not too different to Zelda 64 or Mario 64 (though on a much smaller scale ;-)). I've not really played with 3D, so this feels like enough of a challenge that I'll learn a ton of stuff, but simple enough that I (hopefully) won't just abandon it half-way through (something I may have a reputation for...).

Over the coming months, I'll blog about what I'm doing as I go. I can't promise I won't give up and go back to a 2D Jungle Blocks, but at least if I do, I can say I tried!

giffgaff Earnings - July 2010 - £60 this month!

2010-08-05T13:00:00Z

Those that follow me on twitter have probably noticed me talking about giffgaff a lot lately. This isn't just some scam to try and earn money - I really believe what giffgaff are doing is fantastic. They're bringing lower prices (and cash for referring friends) by saving money by not having expensive TV ads or call centres. But this post isn't about who giffgaff are - you can read that in a previous post.

Payback

To encourage customers to spread the word, giffgaff offers £5 to new customers that are referred by a friend. The friend also gets £5 as a thank you. So everyone wins. Additionally, you can earn money by helping out in the forums or even send emails to your friends about giffgaff. The cash you earn can be paid out as credit or even real cash via paypal!

So, over the last few weeks I've been spreading the love, and earning a few quid. To show how easy this is, I decided to publish my earnings each month to show what you can earn! I'll update this post each month with the latest figures.

July 2010 Earnings - £59.52

Sim cards activated £5.00 x 8 = £40.00
Emails sent £0.50 x 10 = £5.00
Forum participation £14.52
Total: £59.52

Note: The forum participation figure may be different each month, and depends on how much you post. The top posters earn around £28 in a month!

This is a very reasonable amount of cash for doing very little work, and it suggests that not only will I never pay for my mobile usage, I'll probably be able to pick up new handsets for free (or almost free) too!

If all this sounds good, and you want a free sim card with £5 credit to get started (Note: You only get the £5 if you are referred, and you must buy credit to activate the sim card and receive it), then request one here!

Google Wave Notifier 1.9 Released

2010-07-29T20:33:00Z

Version 1.9 of Google Wave Notifier is now available. Unfortunately I didn't get around to adding Google Apps support, because it's turning out to be more complicated than expected, but I'm still hoping to have it done soon. The new features in v1.9 are:

Changed source code over to Mercurial from SVN :)
Added ability to customise the URL used to open Wave to support options like "minimized:nav,minimized:contact".
Reduced minimum poll frequency to 1 minute.
Added code to check for the presence of a network before checking for new waves in the background.

Fore more info, visit the Google Wave Notifier website or go direct to the Google Code project page for the download.

The source code for Google Wave Notifier is available (using Mercurial), so if you're got an idea for some new features, I'm open to acceptable patches!

For further updates on Google Wave Notifier, you can follow @WaveNotifier on Twitter!

How to get Dirt Cheap O'Reilly Books (Legally) on any Device!

2010-07-25T18:48:00Z

Over the last few years, I've picked up a few O'Reilly books on my iPhone from the App Store. It's not particularly fun reading on an iPhone screen (especially pre-iPhone4), but the books are crazy cheap. Like £3 cheap!

I figured that the reason these books were so cheap, was that they're locked to the iPhone, and you can't read them on your PC or other device. So I was pretty made up when I realised that any (most) iPhone apps work sync to the iPad!

Unfortunately, I didn't realise the iPad would render text so nastily at double-size (I'm hoping iOS4 will fix this).

Nevermind. Today, I found a gem on the official O'Reilly site. How to extract EPUB books from those cheap book-apps we've been buying. This will allow you to read them in your browser, or any other device that supports the EPUB format. This means you can read those books in iBooks on your iPad :)

For the less technical, I've put together some screenshots showing how to do this. First, find the app in the Apps section of iTunes. Right-click and choose "Show in Explorer/Finder".

Windows Explorer (or Finder, if you're on a Mac) will open, and highlight the file for this book. Right-click and choose Copy.

Paste the file somewhere temporary, and rename it to end in ".zip".

Double-click the zip file, and navigate to the Payload folder.

Inside the Payload folder, you'll see a .app folder. Navigate into that.

Inside, you'll find a folder called "book". This is the interesting one. Right-click, and choose Copy.

Paste the folder somewhere, and navigate inside it. You'll see three items: A folder called "META-INF", a folder called "OEBPS" and a file called "mimetype". Highlight these files, right-click and choose Send To -> Zipped folder.

Rename the new folder to end with the extension ".epub" and copy it to a folder where you would like to store your books. Then from within iTunes, choose "File -> Add Folder to Library" and select this new folder. Alternatively, you can drag the epub file into iTunes.

That's it! Now your book will appear in the Books section of iTunes, and will sync into iBooks on your iPad. You can also transfer the file to other devices that can read this format.

This makes those very-attractive O'Reilly books look even more attractive!

giffgaff - Free Sim Card with £5 Credit

2010-07-13T21:24:00Z

Since posting about giffgaff, I've had quite a lot of emails request sim cards. I decided to make the process a little easier by adding a form to the page. If you're like a giffgaff sim card with £5 free credit (which you can only get by being referred to giffgaff), then please fill in this form.

For lots of reasons why should you move to giffgaff, see the original post :-)

giffgaff - Owned by O2, Using O2's Network, but Far Cheaper!

2010-07-11T11:26:00Z

Update 27th Aug 2010: giffgaff have added a new £5/month Goodybag which gives 2p/min calls and 1p/sms! No better time to request a free sim!

Update: giffgaff have extended their half-price goodybag deal for another month! No better time to request a free sim with £5 of credit!

When my 18 month iPhone contract on O2 recently came to an end, I was surprised to find that moving to O2 PAYG was far cheaper than moving to their simplicity tariff. Additionally, the simplicity tariff was a 12 month contract unless you paid an extra £5/month. This was a no-brainer. I might want an iPhone 4 in a few months, so a 12-month contract (that is still more expensive) is out of the question. I moved to O2 PAYG.

Only a few weeks later, I seriously started looking at giffgaff, a company I'd been told about by @monkeyonahill. It turns out, giffgaff are owned by Telefonica O2. The company is run independently, but is a virtual operator on the O2 network. This means the infrastructure is exactly the same as you get with O2.

Prices

So, let's get on. The reason giffgaff looked attractive, was the prices. giffgaff don't have call centres, or TV adverts, or other expensive crap. They're lean, and they pass their savings on to their members. This shows in the prices!

Calls and texts between giffgaff members are always free, provided you have topped up within three months
Calls to other mobiles and landlines are 8p per minute
Text messages to other mobiles and landlines are 4p each
Mobile internet is free for everyone until 1st October
0800 numbers are free

If you compare these prices to your current tariff, I'm sure you'll be surprised. But I know what you're thinking - you get inclusive calls/texts, right? Well giffgaff can do that to!

giffgaff sell "goodybags" which last a month, but generally work out cheaper if you use your phone a lot. For July and August, all goodybags are half-price! This means for £5, you get:

100 minutes
Unlimited texts (no, really!)
Unlimited internet (no, really!)

You can also "queue up" a goodybag, so for this month, you can pay £10 and get this goodybag for both this month and next month!

Unlimited? My Arse!

Something that truly annoys me, is the use of the word "unlimited", when it doesn't mean unlimited. In giffgaff's case, it really is unlimited. This may change in the future, but for now, you're not going to get told off for surfing too much or sending too many texts!

Keep Your Number!

As with all mobile networks, you can bring your number over to giffgaff. You don't have to do this straight away - you could request a sim below, and test it out with the assigned number, and then if you decide to stick it out, get a PAC code from your current provider and port your number in!

Interested? Want a Free Sim with £5 Credit?

Because giffgaff is "people powered" and doesn't spend lots of money on marketing, it's up to their members to help them grow. As such, if a member refers a friend, they each get £5! (I got mine thanks to @appleboyy). You also earn money back for other things like helping out in the forums.

If you're interested in trying out giffgaff and would like a sim card with £5 credit for free, then please fill in the form below. You can also request sim cards direct from the giffgaff website, however unfortuantely, you won't get £5 free credit.

Please note: You will need to purchase credit (currently £5 with the half-price goodybag) to activate the sim card. This does not need to be done now - only once you have received the sim card from giffgaff.

For more info, do check out the giffgaff website. If you have questions, pop by the forums. The members are a friendly bunch, and happy to answer your questions!

I really think companies like giffgaff are the way forward. Today people are used to hunting around to save money and we're becoming less reliant on things like TV advertising to find things. Let's hope this is a sign of things to come - spend 10 minutes on Google, save a fortune :-)

More Lovefilm Hilarity

2010-07-10T10:17:00Z

I recently blogged about the epic saga when I tried to change my email address with Lovefilm. It resulted in me closing my account (for security reasons, not out of spite!).

As the adventure ended, I had another few emails from Lovefilm that I thought I should post, because it just further highlights how bad their customer support is. Don't worry, this one is only three emails :-)

It looks lie some wires got crossed, and after someone already solved my issue (closed my account), I got an email from someone trying to do the same.

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny,

Thank you for your email.

We will be happy to investigate this query for you, but at present I'm afraid I am unable to positively identify your account from either the email address you have given "(removed)" (which does not appear to be registered in our database), or from your name as you have supplied it below.

Rather than leaving the issue open, I replied to inform them that everything was sorted.

From: Danny Tuppeny
To: LOVEFiLM Support

Hi,

It looks like the account has already been updated. I was somehow still logged into it, and now my name shows as "Removed Removed" and there is a "videoisland" email address :-)

Danny

I probably could've worded this better, but I thought it was quite clear that the problem had been dealt with. Ofcourse, as I've seen, it doesn't really matter how clearly you explain things to the Lovefilm Customer Service team anyway...

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny,

Thank you for your query regarding your account.

I would be happy to provide you with the login details for your DVD Rental account, however before doing so, please can you kindly provide us with the following details below:

1. Your registered first and last name
2. Your registered delivery address with postal code
3. Your recent titles dispatches on your account.

Once we have these details, we shall be able to perform the necessary security checks and provide you with the necessary login details.

Truly gobsmacking. Needless to say, now my account is removed, these guys are on my blocked senders list. I really don't want to hear from them anymore until the learn to read their emails before replying!

Lovefilm - Massive Customer Service Failure

2010-07-06T16:41:00Z

At the start of this year, I posted that I had decided to abandon my old ".me.uk" domain in favour of tuppeny.com and dantup.com. It's been a long, painful process updating my email address with all the companies I have accounts with.

However, none of the fun I've previously had can possibly compare with the recent communications with lovefilm.com. I really don't understand how any company can possibly get things so wrong, but I'm posting it here to share with the world.

WARNING: This is quite a long post. This really is how hard it is to change your email address at LOVEFiLM.

Here's a short summary:

Danny: Please change my email address.
Lovefilm: Done. We also changed your password to something insecure.
Danny: Thanks. I can't find a way to change my password to something secure.
Lovefilm: It's on the account page.
Danny: It's not.
Lovefilm: It is.
Danny: It's not.
Lovefilm: Oh, your account is not active, so you cannot change your password.
Danny: Can you please change my password to something more secure, since you changed it?
Lovefilm: You can change it yourself on the account page.
Danny: No, it's not there.
Lovefilm: Oh, sorry. It's on the account page.
Danny: It's really not. Can you change either my password or remove my card details?
Lovefilm: You can change it yourself on the account page.
...

I really don't know if my expectations of companies are too high, but this entire process seems to have been far more difficult than it should be. I'm posting all emails in full, minus the sensitive information. This emails are real. Every. Single. One.

Changing your email address should not be this hard!

From: Danny Tuppeny
To: LOVEFiLM Support

Hi, 

Please can you change my email address. Your help pages say it's possible through the "my account" section, however, if it's there, I can't find it. 

Old email address: (removed)
New email address: (removed)

The old email address will soon stop working, so should I ever decide to reactivate my account, it would be good to have a working email address assigned to my account.

Thanks!

It didn't take long to get a response. Though for some reason, they'd reset my password.

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny, 

Thank you for your email.

We are happy to confirm that the registered e-mail address on your account is now changed as '(removed)' and we have manually reset your password to 'lovefilm' (lowercase). We request you to login using this e-mail address and password.

I logged in successfully, and looked for a way to change my password. I tried the "my account" page, the "edit profile" page, and many others. I couldn't find anywhere, so I consulted the help section. The help section suggested there was a "change password" option in the "my account" section. Not so.

From: Danny Tuppeny
To: LOVEFiLM Support

Hi,

I'm unable to find anywhere on your site to change it. I've checked every page under the "my account" section, but none seem to contain a "change password" section.

Is it possible you can send me the URL for the change password page?

A simple request, no?

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny,

Thank you for your recent email.

Please accept our sincere apologies for the inconvenience caused to you.

To change the password click on the "my account" link which you will find on the top right hand corner of the screen. Then click onto "change password". Please enter the password and replace with a new one for security reasons.

This wasn't really anything new, but I checked again. There is definitely no option. I wondered if it was related to my account not being active...

From: Danny Tuppeny
To: LOVEFiLM Support

Hi, 

Thanks for the reply. There is no change password link on my "my account" page. I've read through the entire page, and even searched for "password" in both the page, and in the HTML source of the page. 

I don't know whether it's due to something different about my account (such as not having an active package), but the word "password" certainly doesn't exist on the page. 

Please could you advise? 

Thanks

I wasn't really expecting the response I got. It doesn't really make sense. Why would my subscription have any affect on the account password?

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny, 

Thank you for your recent email.

We apologise for any inconvenience caused to you.

We are sorry to inform you that you would not be able to change the password on your cancelled account. If you are willing to change the password on your account your account status needs to be active.

Ofcourse, I'm not going to reactivate my subscription just to change my password (especially when it was Lovefilm who set it to something insecure)! I thought the easiest thing would be to let them change it...

From: Danny Tuppeny
To: LOVEFiLM Support

Hi, 

Please can you change my password from "lovefilm" to something more secure. 

I did not ask you to change my password, only my email address. You could not really have changed it to anything less secure, and I'm apparently unable to change my password because I'm not actively paying!

This account has my personal and credit card details in it. Please remove this information or change my password.

Thanks.

I thought that would be the last of it. Boy was I wrong!

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny, 

Thank you for your email.

I am very sorry to hear that you are experiencing difficulties whilst logging to your account. Could you please login to your account with the following email address and password. 

Email address : (removed)
Password : lovefilm (lowercase)

To change the password, click on the "my account" link which you will find on the top right hand corner of the screen. Then click onto "change password". Please enter the password and replace with a new one for security reasons.

By this time, I am getting annoyed :-) The response I sent wasn't the most polite I sent, but I just wanted someone to fix my password. This account holds card details and they had changed my password to something stupid.

From: Danny Tuppeny
To: LOVEFiLM Support

Hi, 

This is getting rediculous. Do you guys not read the previous emails being replied to? 

There is no option for me to change my password. Someone from your company confirmed that the "change password" option only exists when you have an ACTIVE subscription. 

I do not have an active subscription. YOUR COMPANY changed my password to the NOT SECURE "lovefilm" WITHOUT MY REQUEST.

This account has my credit card details and personal details.

Either change my pasword to something more secure, or remove my details. I cannot change my password myself - we have already established that - because I do not have an active account. 

It appears to me that Lovefilm do not give a shit about the security of their customers details. 

I'm sorry for the tone of my email, but this is the 5th email I've sent. I never requested my password to be reset, and despite explaining many times that the change password option is NOT on the "my account" page, we seem to be going in circles.

For some reason, I got two responses this time. As you can guess, they were both as helpful as each other!

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny, 

Thank you for your recent e-mail.

Please accept our apologies for any inconvenience caused.

I can confirm to you that, I have made necessary changes to allow you to change the password. To change the password, click on the "my account" link which you will find on the top right hand corner of the screen. Then click onto "change password". Please enter the password and replace with a new one for security reasons.

Of course, I checked. There is still no instance of the word "password" on the page, or in the HTML source. I'm not even sure why they would need to make changes to allow me to change my password. Either way, it hadn't worked. Here's the second email:

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny, 

Thank you for your email regarding your account. I apologise for any inconvenience caused.

I would kindly ask you to refrain from using such expletives when contacting us in the future. 

When you need to change your email address through ourselves, we have to also change your password to make sure you can access the account. This is to make sure that you can access the account and do not need to contact us again in regards to the same query.

Once you have logged into your account, you can go to "my account" and change your password to whatever you want. We do not advise customers to keep the password we have provided them for security reasons.

Well, it looks like our little loop is going to continue. They explained why they changed my password in the first place, but they then repeated the same thing. By this point, I really don't know whether to laugh or cry... I have lost all faith in Lovefilm. I sent yet another reply, hoping this will be the end of it...

From: Danny Tuppeny
To: LOVEFiLM Support

Hi, 

There is still no option to change my password on the "my account" page. I have even searched the HTML source of the page, and there is no instance of the word "password". 

I really don't understand how I can make this any clearer. This is the seventh email. I can send you a screenshot of the my account page (or the HTML source) if you wish. The option is really not there.

If you are unable to fix this, then I demand that you remove all of my address and credit card details from your system immediately. I do not wish to have these details compromised because of the incompetence of your company. 

Needless to say, I will not, ever, by reactivating my lovefilm subscription.

I thought this one was pretty clear. Fix my password, or remove my details. I don't want this to go on any longer. For some reason it seems Lovefilm wanted further confirmation!

From: LOVEFiLM Support
To: Danny Tuppeny

Dear Danny,

Thank you for your recent email.

I am very sorry but you will not be able to change the password on an
inactive account via the website. I can see that we have changed the
password on the account to lovefilm, as you requested that the email
address be updated to your new email address. When we update an email
address then we will change the password to allow you to have access to
your account.

Could you please respond to this email confirming that you are looking
for all your details to be removed from our database.

Well, you can't say I didn't try. The funniest thing about this email, was the signature:

Have we done something that's really made you go 'WOW'?  Why not nominate us for an award?  The WOW awards are the only Customer Service awards based on your nominations.  Go to http://www.thewowawardswebs.co.uk/lovefilm to find out more.

My final email, was of course to confirm that I would like to have my account closed. Lovefilm have lost me as a customer, for life. This was a shocking experience, and I hope it's not commonplace at Lovefilm. In my reply, I summarised the stupid things they'd done, in the hope that someone might care enough to fix things.

From: Danny Tuppeny
To: LOVEFiLM Support

Hi,

Thanks for the reply. I can confirm that I would like you to completely close my account. All my credit card details, address details, email address, etc., should be removed from your system.

I really hope your company cares enough to learn from this experience. I had a simple request to change my email address, and your company failed very badly in many, many ways.

You changed my password without me asking.
You changed my password to something completely insecure.
You changed my password and told me to reset it, despite the fact I am unable to change my password.
It took over 15 emails back and forwards before your company was able to resolve this issue, because none of my emails were read correctly.
The resolution was a lost customer, and a deleted account. It should have been a changed email address and a happy customer.

This is incredibly disappointing, and not something I would expect from a company of your size.

I hope you don't mind that I'm going to post a full account of everything that happened, including all emails sent by both parties, in full, to my blog. This epic adventure needs to be shared.

Thanks,

Danny

I finally got an email confirming my account would be deleted "as soon as possible". While that means nothing to me, I have been to visit (and I was still logged in), and I can see my card details are gone, and my name is set to "Removed Removed" with a "videoisland" email address. finally!

Blizzard Email Fail - Solved!

2010-06-30T09:03:00Z

I recently posted about some failure in changing my email address on various big-name websites. One of those, was Blizzard's Battle.net system, used to login to Blizzard games such as World Of Warcraft. The problem was that an error was occuring, but the error text was not displayed:

This problem has now been fixed, so today I was able to see what the actual error was:

"Your password cannot by the same as your email address"

Now, I know what you're thinking. What am I doing having my email address as my password? Well, I don't. However, if you ignore case, the first few characters were the same. Looks like this was the problem all along.

Now I knew what the problem was, it was easy enough to change my password, and then I could successfully change my email address.

+1 to Blizzard for fixing the error message not displaying, but -1 for not making it very clear. I thought the validation was working against the wrong input boxes for a minute. Better wording would be that "your password cannot contain part of your email address" or similar.

I'm just happy I managed to fix it before having to share the old, defunct address for Real ID :-)

Email Fail: Microsoft, Blizzard, Facebook and more

2010-05-03T17:13:00Z

A few months back, I blogged about changing domains and over the last few months I've been slowly updating email addresses at all the sites I have accounts for to a new domain. Rather than using a catchall this time, I decided to go with plus addressing. This means I can allow catchall mail to be swallowed (which will reduce the amount of spam I see) while still maintaining individual addresses for filtering/blocking purposes.

Microsoft Email Fail

Today I came to update the email addresses associated with my Microsoft passports (Live IDs, whatever they're called this week). My current addresses all end with dantup.me.uk (which won't be renewed), and I wanted to change them to be an address at my new domain.

I found the correct page to change my email address. It was a bit frustrating that I had to type my password (the form doesn't allow pasting), but I can live with that. What I can't live with, is being told I can't use my email address! Note: This is unrelated to using a plus in the email address, since it happens without it.

Unfortunately there's very little information in the error to help me track this down. I have absolutely no idea what the problem could be. If Microsoft are no longer allowing custom email addresses, then the radio option shouldn't be there. If my domain has been somehow blacklisted - I'd love to know why!

Unfortunately the help links all point to stupid FAQs and help pages, with no visible way of contacting Microsoft with these kind of issues. The issue is annoying on its own, but when you're a company the size of Microsoft, not having a way for a user to contact you with these sorts of problems is inexcusable.

Blizzard Battle.net Email Fail

OK, time to try updating my Battle.net/World of Warcraft email address. World of Warcraft accounts were recently merged into Battle.net accounts, so the website is all fairly new. Similar form to the Microsoft one, though it asks for the answer to my secret question. I complete all fields correctly and click Save. Doh! An error. Unfortunately, there's no error text - just a box! Note: This is unrelated to using a plus in the email address, since it happens without it.

I checked on Google and it seems I'm not the only one with this problem. Some people had joy changing browser or clearing caches, but I'm stuck with the problem in IE8 and Chrome regardless. Most of the responses from Blizzard employees in the forums just ask the player to phone them rather than actually trying to track down the issue. Well done, Blizzard.

Facebook Email Fail

OK, Facebook went a little smoother. I could change my email address. What I couldn't do, however, was use a plus in my email address. Just in case you're wondering, here's what the specs say about plusses in email addresses:

Without quotes, local-parts may consist of any combination of alphabetic characters, digits, or any of the special characters

! # $ % & ' * + - / = ? ^ _ ` . { | } ~

Looks pretty clear to me. Plusses are valid. Let's see what Facebook have to say about this in a reply to someone that's already notified them of this bug in their website.

Well, it's quite clear Facebook don't give a shit. "There are no exceptions to this rule". Well that's just super. They may as well have said "We can't be arsed fixing our site, so please change your email address". This kinda of attitude is pathetic from a company this big. What's wrong with "Thanks for the report - I'll pass this on to the correct team and we'll see what they can do"? If you're not going to fix it, at least lie to us.

Other Email Fail

Unfortunately, the email fail doesn't end there. Here's a few other sites/apps/games that don't allow plusses in email addresses I've come across just since January:

Facebook
Dominos Pizza
Square Enix Final Fantasy MMO Beta Signup
Lord of the Rings Online MMO
Aika MMO
Perfect World MMO
Mersey Tunnels FastTag
Swiftcover Car Insurance
Argos online reservation
Xbox Live

I'm sure there are many, many more. I'd like to send emails to them all and let them know (from an email address containing a plus, ofcourse) but based on the Facebook response above, I'm really not sure it's worth the effort.

Is Google Code Readying Git Support?

2010-03-19T18:11:00Z

It looks like Google Code could be working on adding Git support (at last)! I've been following the native Git support issue in the issue tracker and some interesting comments have appeared today:

(8) sussman: Tell you what... as soon as git can do HTTP reasonably, let us know, and we can look into the issue of git on googlecode. :-)
(12) rickvug: Git over HTTP is now more efficient as of version 1.6.6. For more information see http://progit.org/2010/03/04/smart-http.html. I guess this is now something to look into? :)
(13) sussman: We're very aware of this, since it was a googler who wrote the new HTTP protocol. :-)

The fact that Google are "very aware" (not just aware, but very aware!) and one of their own staff has had a hand in this, I wouldn't be at all surprised if Git support showed up soon.

Watch this space! (or, the issue tracker!)

(Solved) Intermittent Server."CreateObject Failed" 'ASP 0177 : 8000ffff' Error Creating .NET COM Components

2010-03-14T11:32:00Z

Over the past year or so we've been wrestling with an intermittent error from classic ASP when trying to instantiate .NET components with Server.CreateObject. Everything works fine 90% of the time, and now and then we'll start seeing this error:

Server object error 'ASP 0177 : 8000ffff' 
Server.CreateObject Failed 
<FileName>.asp, line <LineNumber>
8000ffff

Once this error starts happening, it generally persists until we move the application to another application pool or restart IIS. Recycling the app pool does not fix the problem.

Unfortunately, the error message and number are fairly generic, so it's been pretty tricky to track down. There are lots of people with very similar issues posting all over the web that have solved them in various ways. None of the solutions we found have ever worked for us. Most of them were people seeing the error all the time (eg. permissions errors), but we found very few people seeing our behaviour where the code would just randomly stop working.

After many, many months of searching, we found some reports of similar behaviour being caused by installing an Internet Explorer 7 patch. Many people were rolling the patch back with some success. Rolling back patches doesn't seem like the best thing in the world, so we've always avoided the complications that go with it.

The Solution

Eventually, we found the solution. It's in this Microsoft knowledgebase article KB945701. The problem appears to be a failure to read some IE-related values from the registry. The hotfix on the page above (which is included in the latest service pack, so you probably do not need the hotfix) adds the ability to ignore these errors by setting a registry key:

Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\
FeatureControl\FEATURE_IGNORE_ZONES_INITIALIZATION_FAILURE_KB945701
Note If the FEATURE_IGNORE_ZONES_INITIALIZATION_FAILURE_KB945701 subkey does not exist, you must manually create it.
Right-click FEATURE_IGNORE_ZONES_INITIALIZATION_FAILURE_KB945701, point to New, and then click DWORD Value.
Type w3wp.exe to name the new registry entry, and then press ENTER.
Right-click w3wp.exe, and then click Modify.
In the Value data box, type 1, and then click OK.

After setting this registry key, a simple app pool restart will apply the change. No longer will your .NET COM components randomly stop working with no real solution except shuffling application pools!

Wanted: Artist for iPhone Game

2010-03-09T22:30:00Z

I posted this on Buzz, but thought it worth also repeating here. I'm working on a few iPhone games, but I'm no good when it comes to graphics. I'm looking for someone that might be able to do the graphics for me. One of the games involves a penguin with an umbrella, falling from the sky :-)

Edit: Got a designer signed up. Time to start coding!

Google Buzz - What's it all About?

2010-02-11T22:09:00Z

Like every man and his dog, I've been playing around with Google Buzz the last few days. So far it's looking quite good. It's not entirely accurate to call it a Twitter clone, but it certainly works in a very similar way.

If you'd like to have your blog posts (or other external content) posted on Twitter, you'd need to find a service that uses the Twitter API to post messages. Google Buzz works the opposite way, and Google will pull updates from other services (Twitter, Flickr, Blogs, Youtube, Google Chat, Picasa, Google Reader) and show them in your Buzz feed.

Currently, there's no built-in way to send your Buzz updates back to these services, though there are a few 3rd party services and workarounds to make this work.

DeWitt Clinton posted an interesting Buzz (?) about where Google Buzz is heading. It focuses on being open and interoperable. Rather than a closed system like Twitter, where you can only share with other Twitter users, Google are promoting an open system where different hosts can exchange messages with each other, like how other systems like Email and Google Wave work.

I have high hopes for Google Buzz. For too long have we put up with Twitter's fail whale and erratic API. It's time for their monopoly to end and everyone to play nicely together.

If you're giving Google Buzz a try, you can follow me using the button in the right column of my Google Profile page. If not, you can still find my ramblings on Twitter.

World of Warcraft Addon Updaters - What Do You Use and Why?

2010-01-28T21:21:00Z

This post is for the World of Warcraft players among you. More specifically, those that use Addons. I'm interested in whether you use an addon updater (that is, software to keep your addons up-to-date). If so, which one, and what you do and don't like about it.

I don't want to get into a debate about Curse, WoWInterface and WoWMatrix (that ship has sailed, and it's been "discussed" more than enough), so please try to avoid mentioning that ;-)

Feedback is encouraged in the comments, the World of Warcraft Forums, or tweeted @NewsWoW.

Open Sourcing Google Wave Notifier

2010-01-22T17:44:00Z

Yesterday I finally got around to do something I've been planning on doing for a number of weeks. I uploaded Google Wave Notifier to Google Code. From today, Google Wave Notifier is Open Source!

In hindsight, I should have done this much sooner. The app is now very stable and contains all the features that I planned to implement (and more!). I'm still getting lots of feature requests, and I really wish I could implement them all. However, in reality, I just don't have time. By sharing the code with the world, all of these features need not go unimplemented!

I still intend to work through bugs and some feature requests myself, though it's unlikely to be at the rate of previous releases. With help from the community, hopefully we'll still see regular releases and new functionality.

For more info, or to download the source, please visit the Google Wave Notifier page on Google Code.

Generic 301 Redirection Script for Google App Engine

2010-01-17T11:02:00Z

Although this post is about writing a redirect script for App Engine, it doesn't require that any of the sites are hosted on App Engine, so it could be useful to you even if you're hosting .NET websites elsewhere, but need to handle redirecting old domains.

If you believe what Derek Says, I change my domain every 5 minutes. While this might be a slight exaggeration, I have moved many domains recently and needed to deal with the usual problems this brings: search engine rankings and existing inbound links.

301 vs 302 Redirects

The most common type of redirect you'll see on the web is a "302 Temporary Redirect". This is what most frameworks will output when you redirect (eg. Response.Redirect() in ASP/.NET or self.redirect() in App Engine). The "Temporary" part of this redirect means the redirect is a one-off and will not always be served. This is handy for example, for redirecting a user back to a page after logging in to your site. Since this page may be different each time, the redirect is not fixed.

The other type of redirect, 301, is a "Permanent Redirect". Most frameworks support these redirects without outputting your own headers. Eg. in App Engine you can use self.redirect(url, permanent=True). This type of redirect means the redirect will always occur to the same link, and that clients are free to bypass your script and assume it will always go to the same place. This is the redirect that we want to use when moving domains. It tells search engines that this page has moved, permanently, and they may associate any rankings for the old page, with the new page.

Generic App Engine Redirect Script

So, now we know what type of redirect to use, it's time to build a script to handle our redirects. If you have multiple domains like me, it makes sense to write a script that can handle them all in one go. I've decided to set up a new App Engine app with the sole purpose of redirects for all my domains from this point forward.

Since I recently decided to move everything from dantup.me.uk to dantup.com, I had a few different domains to redirect. blog.dantup.me.uk needs to map to blog.dantup.com, wavenotifier.dantup.me.uk needs to map to wavenotifier.dantup.com and all of the unused domains (eg. dantup.com, www.dantup.com, tuppeny.com, etc.) need to map to the root of my blog.

App.yaml Setup

If we're gong to be redirecting all incoming requests, we need to route all requests through our script. This is why it's important to use a new App Engine app rather than piggy-back onto an existing one. We'll see up our App.yaml file to route all requests into a script called main.py.

application: myapp-redir
version: 1
runtime: python
api_version: 1

handlers:

- url: /.*
 script: main.py

Next we need to define a way to hold all of the data we'll need to perform our redirects. As well as the old domain and the new domain, we need to know whether to map urls from the request onto the new domain, or just redirect to the root. Eg., when I moved my blog from blog.dantup.me.uk, I wanted blog.dantup.me.uk/mypost to redirect to blog.dantup.com/mypost. However I want tuppeny.com/anything to just redirect to the root, blog.dantup.com.

A dictionary seems to be a good way to store this data because we can perform lookups on the domain quickly, and we can store the new domain and a boolean for the url mapping as a tuple.

# Old Domain: New Domain, Map urls (else redirects to root)
urls = {
    'www.dantup.com': ('blog.dantup.com', False),
    'www.dantup.me.uk': ('blog.dantup.com', False),
    'www.tuppeny.com': ('blog.dantup.com', False),
    'dantup-redir.appspot.com': ('blog.dantup.com', False),
    'blog.dantup.me.uk': ('blog.dantup.com', True),
    'feeds.dantup.me.uk': ('feeds.dantup.com', True),
    'wavenotifier.dantup.me.uk': ('wavenotifier.dantup.com', True),
    'wavenotifier.tuppeny.com': ('wavenotifier.dantup.com', True),
    'go.dantup.me.uk': ('go.dantup.com', True),
    'go.tuppeny.com': ('go.dantup.com', True),
}

In addition to this mapping, we should declare a default domain, so if any requests make it to our script that don't have a mapping, we can redirect there. We'll use a 302 and also log and email this, since it's probably a mistake.

DEFAULT_URL = 'http://blog.dantup.com/'

This is all looking a little complicated, so it makes sense to build in a way to test our mappings without having to set up lots of entries in the hosts file. I've decided to declare a boolean that enables/disables testing. When testing is enabled, if you navigate to /test then it will output a bunch of URLs and the locations they'll redirect to. We'll keep a list of URLs to test in the code:

ALLOW_TEST = True

test_urls = [
    'http://www.dantup.me.uk',
    'http://www.dantup.me.uk/',
    'http://www.dantup.me.uk/blah',
    'http://www.dantup.com',
    'http://www.dantup.com/',
    'http://www.dantup.com/blah',
    'http://www.tuppeny.com',
    'http://www.tuppeny.com/',
    'http://www.tuppeny.com/blah',
    'http://blog.dantup.me.uk',
    'http://blog.dantup.me.uk/',
    'http://blog.dantup.me.uk/2010/mytest',
    'http://feeds.dantup.me.uk',
    'http://feeds.dantup.me.uk/',
    'http://feeds.dantup.me.uk/2010/mytest',
    'http://wavenotifier.dantup.me.uk',
    'http://wavenotifier.dantup.me.uk/',
    'http://wavenotifier.dantup.me.uk/2010/mytest',
    'http://wavenotifier.tuppeny.com',
    'http://wavenotifier.tuppeny.com/',
    'http://wavenotifier.tuppeny.com/2010/mytest',
    'http://go.dantup.me.uk',
    'http://go.dantup.me.uk/',
    'http://go.dantup.me.uk/mytest',
    'http://go.tuppeny.com',
    'http://go.tuppeny.com/',
    'http://go.tuppeny.com/mytest',
]

Now we've set the data up, it's time to write the code to handle the redirects. To allow for easy testing, we'll first create a method that takes a URL and returns where it should map to. This will be called by both the tests and the real redirects.

def get_redirect_url(url):
    scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
	
    # Discard any port number from the hostname
    netloc = netloc.split(':', 1)[0]
	
    # Fix empty paths to be just '/' for consistency
    if path == '':
        path = '/'
	
    # Check if we have a mapping for this domain
    if netloc in urls:
        # Grab the redirect info tuple
        redirect_info = urls[netloc]
        # Root redirects
        if not redirect_info[1]:
            return 'http://' + redirect_info[0] + '/'
        # Redirects with paths
        else:
            return urlparse.urlunsplit(['http', redirect_info[0], path, query, fragment])
    # No mapping, so return None
    else:
        return None

This code is fairly straight forward. It uses our mappings dictionary to look up the domain to redirect to, and whether to include the path information. Next we need to write the code that actually handles incoming requests. This will check whether test mode is enabled, and if the request is '/test'. If so, it will output a table using out list of test URLs above. Otherwise it will call the same method, but actually perform a redirect. If we couldn't match a domain, we'll use a 302 redirect to the default URL, and send an email/log.

    def get(self):
        # If we're allowed to test (eg. local), and requested /test, then output the test
        if ALLOW_TEST and self.request.path == '/test':
            self.response.out.write('<h1>testing</h1>')
            self.response.out.write('<table>')
            for test_url in test_urls:
                self.response.out.write('<tr><td>' + test_url + '</td><td> </td><td>' + get_redirect_url(test_url) + '</td></tr>')
            self.response.out.write('</table>')

        # Otherwise, just go ahead and redirect
        else:
            # Perform redirect
            url = get_redirect_url(self.request.url)
			
            if url:
                logging.info('Redirecting ' + self.request.url + ' to ' + url);
                self.redirect(url, permanent=True)

            else:
                # Log that we didn't know what this was, and redirect to a good default
                logging.error('Unable to redirect this url: ' + self.request.url);
                mail.send_mail_to_admins(
                    sender='"DanTup Redirect" <myemail@mydomain.com>',
                    subject='Redirect Script Error',
                    body='Unable to redirect this url: ' + self.request.url
                )

                # Don't do permanent (301), since we don't know what this is.
                # Move it into the dictionary above if needed
                self.redirect(DEFAULT_URL)

There's a lot of code there, but it should be fairly simple to understand. We handle the test mode by just spitting out a table of our test URLs and the redirects. We can then look over this manually to make sure everything looks correct before going live. Otherwise we work out the redirect for the current request and redirect. If no URL was found, we log and email the attempt, and redirect to the default URL. When the email comes through, we can then add the domain we missed to the mappings dictionary and specify how it should be handled.

Naked Domains on App Engine

You'll notice that "naked" versions of my domains are missing from the script. This is because App Engine doesn't support naked domains, so these are all set up as redirects in my registrars control panel. They support 301 redirects with the same URL mapping options (eg. redirect all to root, or copy the path).

Conclusion

It didn't take much to write a simple generic redirect script, and now we can handle redirects for all domains in the future. This simply needs setting up on App Engine and any number of domains pointing at it. It's worth noting that you can point multiple domains from different Google Apps accounts at the same App Engine app. There is no requirement to use App Engine for hosting your sites in order for this script to be used. The fact that blog.dantup.com is hosted on App Engine doesn't change anything. You could redirect to an Azure site if you wished! Though you probably wouldn't want to ;-)

Full Listing

For convenience, you can get a copy of the full script.

Google App Engine Benchmarks - db.put() Performance

2010-01-16T20:23:00Z

Over the past few weeks as I've been using Google App Engine, I've come across people requesting benchmarks so they can compare App Engine performance to other solutions before they try it out. I don't really think comparing Google App Engine and it's Datastore to something like Azure and SQL Server is all that useful (because you'd generally structure things very different on each platform), but either way, it's interesting to see how things perform.

As well as comparing numbers with other platforms, I think it's worthwhile for App Engine developers to know how the different APIs perform (eg. the difference between fetching something from Memcache and the Datastore). Over the next few blog posts, I'm hoping to provide some numbers I gathered on the production App Engine servers. Please bear in mind that App Engine is still considered a "preview" and as things evolve, performance may change (hopefully for the better!).

The first set of data I gathered was on the performance of db.put(), and more specifically, the difference between calling db.put() multiple times with single entities vs calling it with a whole bunch of entities at once. It's very easy to call db.put() multiple times in a single request, but it's usually trivial to change your code to save all the entities in a single call. I thought that illustrating this difference with some pretty graphs might encourage people to use batch operations.

As always, the size and shape of your data will affect your timings. In my sample I used a small entity with only three string properties (and a key_name). You can grab a copy of the data I gathered in CSV format: App Engine db.Put() Benchmarks

db.put() Performance

Each test was run 10 times, and both the mean and median values plotted on a chart. I did this for varying numbers of entities from 10 to 500 (since db.put() has a limit of 500 entities).

As expected, in all cases, the batch method out-performs calling db.put() many times. Both operations scale very linearly (note the first 3 points are not increments of 100, which is why the line doesn't appear straight). For very small numbers (eg. 10 or less) the results are very similar, but as the number of entities increases, it becomes more important to batch up your requests.

It's worth noting that when you call db.put() with multiple entities, they are not combined into a transaction. If one of the writes fails, an error is raised, but any entities that have already been saved are not rolled back. If you want to update multiple entities are part of a transaction, you must do this the usual way by giving them the same parent and using run_in_transaction.

db.put() Performance Consistency

Since each test was run 10 times, I had enough data to draw a chat showing the consistency of the db.put() performance. The closer a line is to being completely horizontal, the more consistent the write performance is.

Individual db.put() Performance

Batch db.put() Performance

As you can see, the calls are fairly consistent, though the more entities you're saving, the bigger the variance. Although from the graph it looks like the batch calls are less consistent, they graph is drawn at a different scale. The batch calls vary my up to 1.5 seconds, whereas the individual calls vary by up to 5 seconds!

I hope to run some more benchmarks over the coming weeks showing the difference between other APIs such as using Memcache to avoid going to the Datastore on every request.

New Domain - DanTup.com - Please Update Your Links!

2010-01-09T12:24:00Z

After much discussion and debate, I've decided to move all of my sites from the dantup.me.uk domain to dantup.com. The .me.uk domain came free with some hosting and was never really intended to become my "main" domain, but it did. It looked like Google were ranking me lower in Google.com vs Google.co.uk because they believed my content to be more relevant to the UK, which isn't really the case.

Over the last few hours I've been slowly migrating things and setting up redirects. An unfortunate side effect is that you may get duplicate posts in my feed (again). Apologies for that, but once everything is moved over, hopefully it's now here to stay :-)

Although not necessary (old URLs will 301-redirect to new ones), if you have any links to any of my sites/feed, it'd be nice if you could update them to the .com to avoid additional redirects. Here's the new URLs:

Blog: http://blog.dantup.com/
Feed: http://feeds.dantup.com/DanTup
Wave Notifier: http://wavenotifier.dantup.com/

It looks like some things that I've moved from one Google service to another (eg. Feedburner to App Engine) are taking some time to make it through the Google network, so the old domains may behave strangle until that's done. Please let me know if you notice anything strange.

Entity Groups, Contention and Transactions in Google App Engine

2010-01-06T17:16:00Z

When I started learning Google App Engine, I misunderstood a fairly fundamental part of the datastore - Entity Groups. The documentation is not very clear, and over the weeks I've seen many questions asked in videos and forums that suggest I'm not the only one that misunderstood. I thought it was worth a blog post to explain.

Entity Groups are not Tables!

The misunderstanding is that people think of entity groups like table in SQL. This is not the case. If you create two entities that are of the same kind, by default, they do not belong to the same entity group. Unless you specifically choose to put them in the same group, all of your entities will be in a seperate entity group. This means if you have 1,000 entities of the same kind, you have 1,000 entity groups containing one entity each! This is not a bad thing, you shouldn't put things into the same entity groups unless you need to, since updates to an entity lock the whole entity group.

Putting Entities in the Same Entity Group

All entity groups have a root entity. You can't put two entities into the same entity group without one of them being the root entity (parent). To put two entities into the same entity group, you set the parent property, like this:

# Create the company
my_company = Company(name='Danny\'s Company')
my_company.put()

# Create the employee
me = Employee(
    parent=my_company,
    name='Danny Tuppeny'
)
me.put()

Because updates lock an entire entity group, you should only use them if you need to. If you're not going to need to update an employee and its company in the same transaction, you don't need to put them in the same entity group.

Entity Keys Include Their Parent Entity Keys

Something worth noting, is that the key for an entity includes the key of it's parent. That means you can do some clever things for performance purposes, such as "Relation Index Entities" as described by Brett Slatkin in this video. Brett creates child entities for querying and converts their keys to their parents keys using the parent() method to then batch fetch the entities themselves.

Hopefully this clears things up a little. If it still leaves questions (or I've missed anything), please leave a comment below and I'll try to update the post.

Profiling Google App Engine with Appstats

2010-01-03T17:49:00Z

As mentioned in my previous post, I've been keeping an eye on my logs after putting this blog live on App Engine. One thing that wasn't so easy to do with App Engine was monitor the datastore calls being made (like I would with Microsoft's SQL Server Profiler).

Luckily for us, Google added some API Hooks into App Engine and with some help from Nick Johnson and Jens Scheffler I managed to cobble together some scripts that allowed me to identify and solve my problems.

Today, by accident, I came across a library that completely blew my hacked scripts out of the water...

Guido van Rossum's Appstats

Guido van Rossum (a Software Engineer at Google, and the author of the Python programming language!) has released a library called Appstats that hooks and monitors the API calls your app makes and presents them in tables and graphs to aid profiling. Setting it up in your app is a breeze (especially if you already use util.run_wsgi_app to run your app).

I got it up and running on my blog via the dev server (though it works equally well on the production servers) to see how well it works. After a few clicks around the app, I navigated to /stats to see what it came up with.

The Appstats Dashboard

The Appstats dashboard is broken into three sections. The first shows stats for each API method you've called. The second shows stats by URL. The third shows recent requests. Each line in these tables can be expanded for a breakdown of the numbers, as shown below.

Request Statistics

Once you click on a request, you'll get a lot of information about the API calls made.

The first thing shown is a timeline showing not only how long each call took, but also when it started and finished (the will help identify large amounts of time being spent outside of the API calls). In the above example you can see that the most expensive part of this request was a RunQuery call. This call fetches comments for a given post when the posts HTML is not in memcache. All other lookups performed are either memcache Gets or db get_by_key_name calls, which are very fast in comparison.

If you expand one of the API calls you'll see a breakdown of what the call involved (both the request and response). You can specify how much text is included in these tables (such as the request/response) by changing the Appstats options.

Finally there's a summary table showing all API calls for the given request, much the same as on the dashboard, though this one includes timings.

Appstats looks to be a very valuable tool for any App Engine developer, and I look forward to seeing what comes out of the team in the future!

Caching App Engine Datastore Results in Memcache

2010-01-01T20:58:00Z

Since moving my blog to Google App Engine a few days ago, I've been keeping a close eye on the logs. This is my first app engine project that's using the datastore, so I wanted to make sure I hadn't done anything silly and I wasn't getting a large number of timeouts. Although it's probably overkill for my blog, to learn the APIs I use memcache to avoid hitting the datastore lots for the same data.

The caching I've implemented is fairly basic. When it generates a page using Django templates, the entire page contents are put into memcache, using a key that includes the page URL. If somebody else requests the same page, I can serve the entire thing from memcache without any processing overhead. This works well because there is no dynamic (different per user) content on any pages (unless you're an admin, but then memcache is bypassed entirely). When a comment is posted or a post is modified, the entire cache is wiped out. This is done to ensure comment counts on list pages are always up-to-date. Since posts and comments happen very infrequently on this blog (a handful per day at most) this shouldn't be an issue.

So, after putting things live, I noticed that some of my page hits were still quite expensive (in terms of CPU) with the caching. It turned out, that when someone visited a page that wasn't in the cache, I had to execute 3 or 4 datastore queries. One to get the Tag or Archive you were viewing (this step wasn't needed for homepage of direct post pages), one for the post(s), one for the tag list and one for the archive list. Since the tags and archive list don't change much, these could be cached across the application, rather than just per-page, which would result in much faster loading of uncached pages (since they would now be only 1-2 datastore hits).

So, I implemented this extra caching as follows, and all was good.

# TAGS: Check cache first
tags = memcache.get('tags')

# TAGS: Get and cache if not there
if not tags:
    logging.info('Saving tags to cache')
    tags = Tag.all().order('name_lower')
    memcache.set('tags', tags, CACHE_TIME)
else:
    logging.info('Got tags from cache')
	
# ARCHIVE: Check cache first
archive = memcache.get('archive')

# ARCHIVE: Get and cache if not there
if not archive:				
    logging.info('Saving archive to cache')
    archive = Archive.all().order('-date')
    memcache.set('archive', archive, CACHE_TIME)
else:
    logging.info('Got archive from cache')

Or so I thought. Although I tested the code worked with no errors, there's no nice "Datastore Profiler" like the SQL Server Profile I'm used to in Microsoft-land, so I assumed everything was good.

Today, it occurred to me that the CPU values for my uncached page views was still pretty high. I was seeing up to half a second CPU time for what should be a single datastore query. This seemed pretty high even for my inefficient coding!

It turns out, I made a rookie mistake. The problem is with this code right here:

archive = Archive.all().order('-date')
memcache.set('archive', archive, CACHE_TIME)

This code does not cache the entities returned by the query. Rather, it caches the query. That means every time I grabbed it from cache and gave it into Django, the query was being executed as the template being parsed!

A quick change to call fetch() forces the query to be executed and now the entities are cached instead.

archive = Archive.all().order('-date').fetch(1000)
memcache.set('archive', archive, CACHE_TIME)

I'm not so keen on the hard-coded 1000, but until I can find a better way to force the query to execute, it'll do the job. My next job is to write some better hooks to allow better profiling of the datastore so I can better test for this kind of issue in future!

Update

After posting in the App Engine group, I was direct to the article Efficient Model Memcaching on Nick Johnson's blog, which shows a better way, avoiding some of the overhead of caching Model instances directly.

Serving Different Ads/Content Based on a Users Location (Geo-Targeting)

2009-12-31T14:55:00Z

You probably haven't noticed, but this blog serves up different ads depending on where you're visiting from. Or at least, it'll serve Amazon UK ads if you're near the UK, and Amazon US ads otherwise. Serving up US ads to UK visitors (and vice versa) is pretty pointless, and I've always tried to avoid showing any ads unless they're relevant and at least targeted to the right country.

There are a number of ways to determine where your visitors are coming from, so I spent some time yesterday trying to find the most reliable way (and preferably one that didn't involve having a huge IP database sat alongside my site!). After much hacking and testing, I found what I believe to be the best way. Google.

Google has a JavaScript loader API, which allows developers to load JavaScript libraries from Google with various benefits. That's not really what we're interested in though, it has something more exciting:

google.loader.ClientLocation

It appears that you do not need an API key to use the JavaScript loader, you can simply reference it at http://www.google.com/jsapi. If you look at the JavaScript served up (which is incredibly fast), you'll see something like this:

google.loader.ClientLocation = { "latitude":50.123, "longitude":-2.876, "address": { "city":"Liverpool", "region":"Merseyside", "country":"United Kingdom", "country_code":"GB" } };

Not only do you get the country, but you get the county, city and even lat/lon pair. For me, the location given was within 2-3 miles of where I live, so if you wanted, you could really localise your ads!

On this site, the country is just sent to a script that will serve up some ads based on keywords I've tagged against a post. You might wish to be a bit more exciting and show your users places or people nearby. This could be especially useful for mobile applications/sites, though be sure to read any associated terms and conditions before using it!

Google Wave Notifier Now in 15 Languages!

2009-12-31T14:50:00Z

When I started writing Google Wave Notifier last month, it was really only for my own use. I wouldn't use Google Wave if I had no way of knowing when I had messages (and keeping a browser open is not an acceptable solution!) so I hacked something together to login to Wave and parse the JavaScript objects.

I decided others might benefit from the app, so I cleaned it up a little, gave it an icon and published it on the web. I had no idea that it would get close to 10,000 downloads in little over a month! Nor did I expect so many people to send in translations!

So here we are 6 weeks down the line, and the app has been translated into 15 different languages!

English
Czech
Danish
Dutch
French
German
Hungarian
Italian
Polish
Portuguese
Romanian
Russian
Spanish
Swedish
Turkish

So I'd like to say thanks to everyone that's helped; using, translating and finding bugs in the app. What was a small script written for personal use has turned into something quite useful!

For more information or to download, see the Google Wave Notifier website!

Redirecting Requests from appid.appspot.com to a Custom Domain

2009-12-30T12:31:00Z

If you're running your app engine project on a custom domain (like this blog), you're probably not so happy that people can still access your app at http://appid.appspot.com.

When I started working on my blog, I realised this might be an issue, and did some investigating into how I could stop it. I found solution on SackOverflow that seemed to do what I needed, so I set it up and got it working.

Not long after implementing this code, I found a few problems:

SSL is not supported on custom domains
Cron jobs fail when Google invokes them with an appspot.com address and you serve a 301

So with some tweaks, I've managed to get this working as required. The only annoyance is the hard-coded '/admin' check. This is to support cron jobs, task queues etc., which are all protected ("login: admin" in app.yaml). They must work with an appspot.com address, because Google doesn't seem to follow the redirect when invoking them. It's possible you could do an IP address check here, but I'm not sure how consistent cron/task queue IP addresses are.

The code is called like this:

def main():
    dantup.run_app([
        ("/\d*", RootHandler),
        ("/feed", FeedHandler),
        ("/tags/.*", TagHandler),
        ("/archive/.*", ArchiveHandler),
        ("/.*", PostHandler)
    ])

And dantup.py looks like this:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

def run_app(url_mapping):
    application = webapp.WSGIApplication(url_mapping, debug=True)
    application = redirect_from_appspot(application)
    run_wsgi_app(application)

def redirect_from_appspot(wsgi_app):
    """Handle redirect to my domain if called from appspot (and not SSL)"""
    from_server = "dantup-blog.appspot.com"
    to_server = "blog.dantup.me.uk"

    def redirect_if_needed(env, start_response):
	
        # If we're calling on the appspot address, and we're not SSL (SSL only works on appspot)
        if env["HTTP_HOST"].endswith(from_server) and env["HTTPS"] == "off":
		
            # Parse the URL
            import webob, urlparse
            request = webob.Request(env)
            scheme, netloc, path, query, fragment = urlparse.urlsplit(request.url)
            url = urlparse.urlunsplit([scheme, to_server, path, query, fragment])
			
            # Exclude /admin calls, since they're used by Cron, TaskQueues and will fail if they return a redirect
            if not path.startswith('/admin'):
                # Send redirect
                start_response("301 Moved Permanently", [("Location", url)])
                return ["301 Moved Peramanently", "Click Here %s" % url]
	
        # Else, we return normally
        return wsgi_app(env, start_response)

    return redirect_if_needed

Hopefully you may find this useful. If you encounter any problems with it, please let me know!

1-2-3-Testing. Is This Thing On? New Blog!

2009-12-29T17:54:00Z

There's nothing like testing a system after it's gone live!

After lots of frantic hacking over the last few weeks, the blog engine I've been writing for Google App Engine is up and running. It's not complete, but the important things (frontend, feed, posting, etc.) are done so it's at least usable.

Apologies if you just got a flood of old messages in your feed reader. I did investigate keeping the Blogger IDs to avoid this, but it seemed more hassle than it was worth, so I'll forgive you for clicking "Mark All as Read" this one time ;-)

I'll post more info (and code) on the blog as I get time, but now I'm off to sort some food out after a hard day's hacking!

If you notice anything messed up in the transition, please do let me know!

From .NET to Python on App Engine: Building a Blog on Google App Engine

2009-12-24T21:50:00Z

Over the past few weeks I've been playing with Google App Engine. I find the best way to learn a new language/framework/platform is to just jump in and write something in/on it. So that's what I'm doing. I've decided to write my blog in Python for Google App Engine.

As I may have blogged about in the past, I wrote a blog engine in Microsoft ASP.NET MVC not so long ago with the aim of moving away from Blogger. It was around 90% complete when I abandoned it for a variety of reasons (one being Azure pricing).

It's entirely possible the Google App Engine blog engine will also be abandoned, but since the hosting is free it at least stands a good chance of seeing the light of day! It'll also make an interesting comparison to the ASP.NET MVC version.

I started writing code a few nights ago, and currently the blog stands at 159 non-blank lines. I'm actually quite impressed with how little code I've had to write to get up and running. Currently there's no back-end, but the displaying of posts, comments, tags and archives are all working. Here's a quick screenshot to prove it exists! :)

Over the coming weeks I'll blog about how I've built it (including code), the pitfalls and the the experience of moving from .NET and C# to Python and App Engine!

Downloading/Exporting App Engine Logs

2009-12-20T16:53:00Z

Although I've been playing with App Engine for quite a few weeks now, I only found out yesterday how I can download the logs from App Engine for parsing locally. There's no export option in the dashboard, nor any option in the Windows launcher. However, you can do this yourself with appcfg.py.

If, like me, you've only used the launcher up until now, and never done anything with Python outside of GAE, you might find the thought of running python scripts a little scary. Don't worry, it actually turns out to be very easy. I'm hoping when you installed the Launcher you ticked the "Add to PATH" option... :o)

The command you want to run looks something like this:

appcfg.py request_logs appname/ output.txt

You should replace "appname" with the name of your app. This is the folder the contains the app.yaml file for the app you wish to get logs for. That means you need to be in the correct directory (or provide a full path). Output.txt is obviously where the logs should be written.

This command will retrieve everything from the last day. This might not be what you want, so you can increase this with the num_days argument.

appcfg.py --num_days=5 request_logs appname/ output.txt

Note that you can also specify 0 to get all logs:

appcfg.py --num_days=0 request_logs appname/ output.txt

And finally, if you want all logs that you haven't already downloaded, you can use --append, which will scan the last line of the existing file, and download anything since:

appcfg.py --append request_logs appname/ output.txt

However, you'll find this doesn't work on Windows, and you'll end up with a load of duplicate entries. This is in the bugtracker, but I don't know if/when it'll be fixed. For now, I'm having to just download everything (and I'm not sure if this is counting towards my bandwidth quota!).

Another option worth noting, is include_vhost, which will include the hostname used, so you can seperate requests for different versions of your app (or different custom domains). You can use this like so:

appcfg.py --num_days=0 --include_vhost request_logs appname/ "Logs/Logs.txt"

And that's all there is to it. Now you can create nice pretty charts in Microsoft Excel (since Google Spreadsheets sucks!) showing how well (or badly, in my case) your app is doing!

Increased App Engine Quotas, for Free?

2009-12-19T13:45:00Z

While writing my comparison of Azure and App Engine pricing yesterday, I had a thought about how to increase some of your quotas without actually paying anything. I'm not sure whether Google would consider this "gaming the system" and stamp on you (and I certainly have no need to do it with my traffic levels), but I thought I'd post it in the interest of sharing.

If you check the App Engine Quotas page, you'll see the differences between the free quota, and the billing enabled quota. For example, if you don't have billing enabled, you may serve 1,300,000 requests per day. If you have billing enabled, you get 43,000,000 requests per day (other limits, such as CPU/bandwidth still apply).

So, if you've taken a look at how the billing works, you'll know that you get to decide exactly how your budget is split between resources. The minimum budget you can set is $1/day.

Consider what would happen if you enabled billing, but assigned the whole budget to something you know you won't exceed (eg., if you don't use memcache, assign it to memcache). Your app will be given the higher "billing enabled" quotas, but it won't cost you a penny!

As I said earlier - I've no idea whether Google will frown upon this behaviour, so if you do it, it's at your own risk!

Microsoft Windows Azure vs Google App Engine: Pricing

2009-12-18T14:40:00Z

As a .NET developer, I was quite excited to hear about Windows Azure. It sounded like a less painful version of Amazon's EC2, supporting .NET (less painful in terms of server management!). When I saw the pricing, it didn't look too bad either. That was, until I realised that their "compute hour" referred to an hour of your app running, not an hour of actual CPU time. Wow. This changes things. To keep a single web role running, you're looking at $0.12/hour = $2.88/day = $20.16/week = $86.40/month.

Anyone that's bought hosting for a small site/app recently will know that this is not particularly cheap!

So, recently I've been playing around with Google App Engine. It has this massive problem called Python (and an even bigger one called Java ;o)), but it's such a nice framework/engine to work with that I've somehow overlooked this and started coding with it. There's so much to like about it. Everything is so simple to deploy, and it scales "out of the box". Want Cron jobs? No worries, specify them in a file in your app, and when you deploy, App Engine will pick them up and schedule them. Want to queue up work to process later so that your pages return faster? Task queues do just that. What's more, you get a ridiculous free quota every day. It may be Python, but this sounds tempting, no?

So, I thought it'd be interesting to compare the costs of App Engine vs Azure. I understand this isn't really a like-for-like comparison, but both can achieve the same sort of things, and while all programmers will have a preferred language/framework (I'm no exception), many can be swayed by a cool framework or hosting.

First off, let's compare what you get for free. Bear in mind that Azure is free until the end of January, but since this is a CTP and won't end soon, I'm going to exclude it. Google's free quota currently has no time restrictions.

	Windows Azure	Google App Engine
CPU Hours	-	6.5hrs/day
Bandwidth (out)	-	1GB/day
Bandwidth (in)	-	1GB/day
Storage (DB)	-	1GB
Storage Transactions	-	10,368,000/day

Well, for free, I think we have a clear winner. If you can run your website/app within the limits above, then you can do it for free with Google. It's worth mentioning that Google let you have 10 apps per account (though you may not balance a single site/app across app instances - they are specifically for separate projects).

But what if you're bigger than that? What if you get Slashdotted or Dugg a lot? You might find you quickly break out of the free limits. How do prices compare?

	Windows Azure	Google App Engine
CPU	$0.12/hour	$0.10/hour
Note that MS bill "per hour app is running" whereas Google bill "per CPU hour consumed"
Bandwidth (out)	$0.15/GB	$0.12/GB
Bandwidth (in)	$0.10/GB	$0.10/GB
Storage (Files)	$0.15/GB/month	Pricing unavailable (Blobstore)
Storage (DB)	$9.99/month (SQL Server up to 1GB)	$0.15/GB/month ($0.005/GB/day)
Storage Transactions	$0.01/10,000	?
I can't find any prices for Google App Engine storage transactions, so it's possible there is no charge (though a limit of 140,000,000/day applies)

Well, that's interesting. I was going to try and calculate at what point Azure would become cheaper, but looking at those prices, it just isn't going to happen. Now it's worth pointing out that not all of the comparisons are fair. Google bill per actual CPU hour (so if nobody visits your site, it's not costing you), whereas Microsoft are billing for each hour your app is live and able to respond. There's also a significant difference between SQL Server and App Engine Datastore (and depending on what you're doing, one will have advantages over the other).

I really hope Microsoft re-evaluate their pricing for small apps. It's too expensive to play around with small prototypes at those prices, whereas Google's offering will let me get started completely free, until my app is churning a considerable amount of traffic, and even then, it'll work our cheaper for the same processing/transfer.

Sorry Microsoft. I love .NET and Visual Studio, but Google App Engine is just so easy and cheap that it's going to be my "toy of choice" for my hobby coding for the immediate future!

Should I move my Blog to Google App Engine?

2009-12-13T17:10:00Z

Over the years, I've written (and re-written) many blog engines with the intention of hosting my blog on my own code. I'm a programmer, that's what we do. We don't like the thought of other people writing our HTML!

If you've been following my blog over the last twelve months, you'll probably know what my language/framework of choice would be when writing a web app, based on my previous posts...

As a .NET developer by trade, I'm experienced with C#, .NET, ASP.NET MVC etc. and it's always been the logical choice. There's nothing I like to write more than C#/.NET

So why are you hosted on Blogger?

An interesting question :) Unfortuantely the answer is not interesting. The answer is simply: Hosting. Windows hosting is a pain in the ass. I've had many bad Windows hosts that have dodgy control panels, poor performance or a lack of required features. Most of this is addressed with IIS 7 because we can now do most things via XML config files, but it still leaves one issue: Price. Prices seem to vary wildly, and it's such a pain to set up that I'm often put off signing up in case it turns out to be a lemon. This is the reason I have a fully working .NET MVC blog engine ready to roll in my Documents folder that's never made it to the big cloud in the sky.

What about Azure?

A few months ago I was really excited about Azure. It looked ideal for what I wanted, and looking at how EC2 is priced, I suspected it would be cheap if you're only getting a few hundred visitors per day. Then Microsoft announced the prices... To run a no-to-low traffic blog would cost more than getting a virtual server! On top of that, it seemed quite hacky to get MVC working, and it's still only in preview.

Enter, Google App Engine

While playing around in Google Wave, I downloaded the Google App Engine SDK and had a play. While Python isn't exactly my language of choice (I've never written it before!), it struck me as quite a good deal. Five million page views per month... for free! It's hosted on Google's infrastructure and scales well, and there's no way I'd hit those limits anytime soon.

So, now I'm having thoughts... Do I become a hacky Python programmer, putting aside all the .NET experience I have, for free hosting with Microsoft's biggest competitor?

If anyone has any experience with running blogs on Google App Engine, I'd be very interested in hearing from them. Should I run with it, or should I stop being tight-fisted and just pay for some Windows hosting?

To be continued!

Google Wave Notifier now in Spanish, Russian, Swedish

2009-12-05T11:47:00Z

With the help of some users, yesterday I added support for Spanish, Russian and Swedish languages to Google Wave Notifier. All parts of the interface are translated and I'll keep adding languages if people are prepared to help with the translations :)

If you'd like to help translate into your native language, please visit the translation page.

Google Wave Notifier Update

2009-11-28T23:45:00Z

Apologies if this is all I seem to be blogging about lately, but it's the only interesting thing I've been doing!

I've just pushed out a small update to my Google Wave Notifier application (v0.6). You can read about it here.

New features include the ability to play the Windows "New Mail" sound when you get new messages, the ability to change the interval it polls for new messages and some general tidying up and bug fixes!

Google Wave Notifier Update - Automatic Update Checking

2009-11-20T13:46:00Z

I wasn't planning on pushing out an update to my Google Wave Notifier application so soon, but I realised that without automatic update notification, people will just get stuck on the old version and never update.

So, I've pushed out v0.4 which will notify you when an update to the app is available. Other than that it just has some minor tweaks (such as a busy icon while it's checking).

If you find it useful, let your friends on Wave know about it!

Click here for the Google Wave Notifier website.

Google Wave Notifier v0.3 Public Release

2009-11-20T11:09:00Z

Edit: Google Wave Notifier has been released now. You can find the latest posts here.

After some thorough testing, I've decided to publicly release the Google Wave Notifier I've been working on recently. Thanks to everyone that helped test it over the last few days.

Because Blogger doesn't support attachments, I've created a download page on Google Sites.

As always, please post your suggestions or bugs to the bug tracker. I hope you might find this useful!

Click here for the Google Wave Notifier Download Page

Balloons! - The World Just Got Smaller!

2009-11-19T20:57:00Z

Note: As I post this, Balloons! has been reduced to £0.59 for the paid version. Get it while it lasts!

A few weeks ago I was lucky enough to participate in the beta of Balloons! for iPhone. Since it's now available to buy, I thought I'd share my thoughts on the app here.

Balloons! is purely for entertainment, though it doesn't really fall into the category of games. The basic idea is that you pick a balloon and attach a message and/or photo to the label, and launch it into the air. The balloon will travel around the globe (electronically, of course) to be caught by other Balloons! users. When someone catches your balloon, they may attach their own message and photo and relaunch it into the sky for someone else to catch.

I've included a few screenshots below, and although you can see the level of polish Balloons! has, unfortunately you can't see the subtle animations that really bring the app to life. The official website has a video that might give you a better idea of how it looks.

The Balloons! menu screen.

When you launch a balloon, you get to choose how it looks from a selection ranging from normal round balloons to heart and bear-shaped balloons.

Catching a balloon is easy. You can catch balloons from all over the world, though realistically, they have to have been travelling for some time to get from one side of the globe to the other!

The Balloon Tracker allows you to see all balloons you've launched, along with the messages and photos that have been attached to them. This is only available in the paid version.

If you click on a balloon in the Balloon Tracker, you get to see the original message/photo, along with any other messages added to it.

Paid vs Free

Like most apps on the App Store, there's a free version and a paid version of Balloons! With the exception of ads, the only difference is the "Balloon Tracker", which lets you see all balloons you've launched, and the messages that have been added to them. It's well worth paying for this additional feature, or you'll be left wondering whether anyone has caught and added messages to your balloons!

Wanted: Beta Testers for Google Wave Notifier Application

2009-11-19T18:31:00Z

Edit: Google Wave Notifier has been released now. You can find the latest posts here.

Those of you lucky enough to have a Google Wave account have probably noticed there's no official app to notify you of new waves. Google Talk doesn't do it, GMail Notifier doesn't do it, and there's no official Google Wave Notifier.

Yesterday I put together a quick app that will check for new messages in waves and light up a system tray icon (along with showing a notification balloon) when new messages are found. Before I release it into the wild, I'm trying to find a few people to help me test it. If you've got a Google Wave account and would be happy to try it, please send me a wave: danny.tuppeny @ googlewave.com.

Derren Brown - How (not) to Beat a (fake) Casino

2009-10-21T19:19:00Z

I've just finished watching a recorded "Derren Brown - How to Beat a Casino". Yet again, I've been thoroughly disappointed :( It's another episode where he's moved away from the mind tricks for which he became famous, and turned to magic and camera tricks.

For those who didn't watch it, Derren "stole" £5,000 from a viewer and aimed to enter a casino and bet the whole lot on a roulette wheel, with odds of 1 in 37, to win £180,000. By calculating the speed of the wheel and ball, he was going to work out where the ball would land in time to place a bet. He had a hidden camera up his sleeve so we could watch, and the whole thing went out "live". Unfortunately Derren guessed the wrong number, and lost the £5,000. He promised he'd give the £5,000 back to the viewer.

The whole thing was a scam. It was not broadcast live.

Early on in the show, Derren pointed out people have electronic devices to calculate the speeds and guess where the ball will land. Unfortunately they're easily detected, so he would have to do it in his head.

Hold on a minute - he had a camera up his sleeve!

It's quite clear that if you can't get away with a small electronic device, you won't get away with a camera! So what really happened? Well, I don't think it's hard to figure out...

The casino was fake.

There's no way he'd get in with a camera, and the shots were so bad there's no way we would be able to tell whether it's a real casino. The film was not live. Derren pre-recorded the bet, taking multiple shots until the ball landed on the number next to his prediction. They just cut the shots in to make it look live, but it was actually all pre-recorded. He got the wrong number on purpose. If he'd got it right, he'd have to pay out £180,000 to the viewer whose money he used. By getting "the next one" he simply gives the £5,000 back and everyone thinks he's genius because he was "only one number out".

Another promising show, ruined by camera and technology trickery rather than the clever mind tricks we all like to see from Derren Brown.

Aion - How Not to Launch an MMO

2009-10-10T19:12:00Z

As is becoming tradition when a new MMO launch gets lots of hype, BootBlock and I recently started playing Aion. Aion had already been out in Korea for almost a year, so the game should be stable and the launch should be smooth. The gameplay videos looked cool - a lot like Final Fantasy XI, the first MMO I ever played (on a console!).

Although the game hasn't really lived up to the hype, and has had more bugs and missing features than NHibernate, it's been an interested change from playing World of Warcraft. Unfortunately, the launch hasn't been anywhere near as smooth as it should've been.

Note: I've tried to use the word "realm" in place of "server" to refer to a named world that you select to create your character on when playing an MMO to avoid confusion with physical (or virtual) servers.

Realm Queues

The first problem Aion has experience is massive (2 hours plus) realm queues caused by too many people trying to log onto the same realm at the same time. Queues of this length are unacceptable for a game that many will only want to play for a few hours after work/school. This may sound like NCSoft just didn't anticipate the demand for their game, but I disagree. There were over 400,000 pre-orders. NCSoft knew this. They also knew exactly how much stock they'd shipped to retailers for launch week. They'd also already launched in several countries - there's no excuse for getting the capacity so wrong.

Scaling in the Wrong Direction

If there's something to be learned from previous MMOs, it's that the population on a realm is a critical factor in how well people can play the game on that realm. Too few, and people won't be able to find groups. Too many, and the realm will be too crowded and people will have to fight over mobs.

Aion came up with a perfect solution. They call them "Channels", and it allows multiple "instances" of a given zone to split the players. While there are some issues with the implementation, this drastically simplifies the problem of managing population. The number of "Channels" can be changed per zone, on the fly, by NCSoft. This means instead of having lots of realms at launch and potentially having the population fall after the first few weeks, NCSoft could simply load a lot of channels on a small number of realms and reduce them over time as the number of players falls (along with the hardware and player caps associated with the realm).

The problem in Aion, is that it looks like their realms are restricted to a number of players (and an amount of hardware), full stop. It doesn't look like the hardware assigned to a realm can be scaled at all. This means when the queues became very long, they were seemingly unable to increase the hardware assigned to some realms (and therefore the player caps on those realms), and instead had to set up some new realms. When the population falls (which it will, after this launch!) they may end up with many underpopulated realms tying up lots of hardware. This may result in some realms being merged - a complicated and messy process.

Character/Realm Transfers

Ok, let's forgive NCSoft for not being able to anticipate demand for their game. They rolled out some new realms pretty quickly, so at least now everyone can play, right? Wrong! Since most of the people playing have created (and levelled) characters on the original (crowded) realms, they'll have to start all over again if they move realms. This means the new realms are great for new players, but not so great for existing players (specifically those that pre-ordered and had already played for over a week when the new realms arrived). Since there's currently no way to transfer your character from one realm to another, you really are stuck between starting again, or waiting many hours every time you want to play.

NCSoft do say that they're working on a character transfer feature for the coming months, but it's really too late. It's an important part of balancing the population of your MMO and should've been there from the start.

Bots, RMT and Spamming

Most MMOs have been at some point plagued by bots and spammers. RMT stands for "Real Money Trading" which refers to trading real money for in-game items. In most pay-to-play MMOs this is strictly against the rules. RMT websites take advantage of players that are happy to spend some real money in order to have an advantage in the game. They will write bots to login and play the game, killing mobs and looting money and items. They will also spam logged in players with messages asking them to visit their websites to purchase gold.

Not so long ago, this was a huge problem in World of Warcraft. However, if you login today you'll probably see very few bots or spammers. Players shouted about this a lot in the WoW forums and Blizzard (eventually) reacted. There are now very easy methods in game for reporting players as bots and spammers. When someone is reported a number of times, they will automatically be silenced and flagged for a GM to investigate. There are also restrictions stopping trial (free) accounts from speaking in public channels or sending whispers. This has had a significant impact on the RMT business and it's rare to see bots or receive spam these days.

Aion launched without any protection against this. The forums are now full of complaints about bots and spammers. NCSoft have patched in a method for blocking users that send you spam messages, but whether enough of these blocks will silence an account is unknown. If not, then it's useless. No spammer sends multiple messages to the same person from the same character. Never mind, NCSoft is on the case with some bans...

Mass Account Banning

In order to quickly reduce the number of bots and spammers in game, NCSoft have started mass banned accounts and IP ranges. It looks like they're hitting a lot of innocent people with the ban hammer as they go. Whilst I don't believe everyone crying in the forum is innocent, there are far too many of them for me to believe they're all botters/gold buyers.

The mass banning has only increased the amount of complains in the forums. The bans happened on a Friday evening, right before the weekend when most people will want to play (and presumably when they have reduced staff to resolve these incorrect bannings).

Aion had a lot of potential to take a huge chunk of the MMO market from Blizzard. Unfortunately a lot of the issues during launch will have had a detrimental impact on the game. Just like Age of Conan, a lot of players will leave after the included 30 days, never to return. Hopefully other companies working on future MMOs have been watching this launch (and others) carefully to avoid making the same mistakes.

Importing iPhone Dev Keys on a new Mac

2009-10-09T12:45:00Z

When I set my Mac Mini for iPhone development, I was told to backup a key file (.p12) because if I lost it and needed to reinstall, I wouldn't be able to deploy to my iPhone (yikes!). I backed it up as suggested, and made sure I had copies of it all over my Mac, PC and the interwebs.

With a clean install of Snow Leopard, it was time to import this keyfile and make sure everything still worked. As instructed in the docs, I double-clicked on the .p12 file and was asked for the password. I entered it and got the following message:

An error has occurred. Unable to import an item. The contents of an item cannot be retrieved.

Oh, shit.

I tried multiple times to import, and even imported other keys I'd exported before it. I'm 100% sure I was putting the password in correctly. No joy, just lots of errors. Although the key appeared in the list (after apparently failing), Xcode refused to recognise it and I'm unable to deploy to my iPhone. This was looking pretty bad...

Google to the rescue!

After a bit of searching, I came across a post from someone having the same issue, and a workaround. Importing via the terminal apparently works. I gave this a shot, and all is good. I can now deploy to my iPhone again. Thanks Dave K!

Just in case you're reading this with the same problem and the link above is now broken, here's a copy of the solution:

Work-around provided by Dave K.

You can use the 'security' command line tool:
security import priv_key.p12 -k ~/Library/Keychains/login.keychain
security import pub_key.pem -k ~/Library/Keychains/login.keychain

I'm Blogging... From a Mac!

2009-08-07T19:42:00Z

I'm shaking my head with shame. I never thought this day would come. I'm writing a blog post... from a Mac!

Those that know me will probably know I'm a massive Microsoft fanboy. I love .NET, XNA, WPF, Visual Studio, Vista and all the other things that come from Redmond. I don't like Apple.

Last year I bought an iPhone. I just couldn't help myself. I've had many Windows Mobile devices from iPaqs to XDAs and they suck. Big time. The iPhone came along and it just wiped the floor with anything out there, so I had to get one. I love it!

Months on, and I've released an Xbox game called Jungle Blocks using XNA. The whole process was pretty awesome. I got to write it in Visual Studio using C# and for very little money my game was out there being played by other people. They were paying money to play my game. That was a cool feeling!

Then my mind started to wonder... Maybe I should dabble in iPhone development? I could release iJungle Blocks! (Ok, maybe I won't call it that). In a lot of ways the App Store seems to work very similar to XNA and the Indie Games and since I'd already filled in all the forms and got an ITIN I figured I didn't have much to lose trying.

Well, except my pride. It turns out that Mr Jobs doesn't want you making money from his iPhone if you don't have his computer. You can only develop for the iPhone on a Mac (legally). This was a bummer. Macs stink. They won't do .NET and they certainly don't do Visual Studio. It was time to do some research!

I started browsing the web and speaking to a few people to find out what coding on a Mac would be. This is what I found out:

The cheapest new Intel Mac is £500
Xcode (the Mac IDE) is crap. It does not have all the features of Visual Studio 2008 :(
Objective-C is verbose and long-winded. There is no Garbage Collection!
All my friends will laugh and point if I buy a Mac

Things were looking grim, but I decided to go nuts. This week, I bought a Mac Mini. After a few hours I discovered a few more things:

Mac keyboards suck. Thankfully my Microsoft Keyboard/mouse works :)
I can't type " @ or #. They're just mapped wrong.
The Mouse on a Mac starts slow and then accelerates veryquickly. It's a pain in the ass.
Alt+Tab is Windows+Tab. Ctrl+C is Windows+C. etc. This is madness!
The Home and End keys don't work as expected. Neither do Ctrl+Left or Ctrl+Right.

I got Xcode installed and started playing around to see how bad this thing was. After a few nights of wrestling with Xcode, Objective-C, Cocoa-Touch and Mac OSX here's what I've found out:

Xcode is not crap. There's still time, but Xcode is not frustrating the hell out of me like I thought it would. It's not all that bad. Nowhere near as bad as some of the crap I've had to use (I'm looking at you, FlexBuilder). I might just get on with it!
Microsoft have Mac drivers for my Microsoft Keyboard and mouse. They not only fix the crazy mouse acceleration, it includes a proper mapping to fix " @ and #!
Even on the highest setting, the mouse still moves too slow on a Mac. The max setting on a Mac doesn't come close to the Max setting on a PC.
Cocoa-Touch is actually really nice :)
There is no nice fix for Alt+Tab. I can swap Windows/Alt keys, but then Copy/Paste becomes Alt+C/Alt+V instead of Windows+C/Windows+V.
These is seemingly no fix for Home/End/Ctrl+Left/Ctrl+Right

So far, things are not as bad as I imagined. And the Mac does have some nice stuff. But will it tempt me away from the lovely world of Visual Studio and C#? Unlikely. It's not as bad as I expected it to be, but it's just not the same experience as developing on a PC.

ASP.NET MVC HandleError Attribute, Custom Error Pages and Logging Exceptions

2009-04-17T18:24:00Z

I'm sure I don't need to tell you how bad serving a Yellow Screen of Death to your users is. Nonetheless, it seems to be pretty common practice across the web. One of the first things I do when setting up a new ASP.NET project is set up custom error pages and ensure all exceptions are logged (who wants to find out about their errors from their clients?). Since things work slightly differently in ASP.NET MVC I thought I'd dig in and find the best way to do the same thing.

The HandleError Attribute

The HandleError attribute (which appears on the default controllers in an MVC project) tells the framework that if an unhandled exception occurs in your controller that rather than showing the default Yellow Screen of Death it should instead serve up a view called Error. The controller-specific View folder will be checked first (eg. Views/Home/Error.aspx) and if it's not found, the Shared folder (Views/Home/Error.aspx) will be used.

But How Do I Log Exceptions?

You might've spotted the problem with HandleError. It just outputs a view, and doesn't let you run any code. This might be fine if you don't want users to see errors but don't really care for fixing them. Hopefully you think this isn't acceptable and you want to investigate all exceptions!

The OnException Method

The System.Web.Mvc.Controller class contains a method called OnException which is called whenever an exception occuts within an action. This does not rely on the HandleError attribute being set. If you're being a good coder and have your own base Controller class you can override this method in one place to handle/log all errors for your site. You might choose to send emails and/or detect duplicate exceptions and discard them. For now, I'm just going to write them all to a text file in my App_Data folder.

protected override void OnException(ExceptionContext filterContext)
{
	WriteLog(Settings.LogErrorFile, filterContext.Exception.ToString());
}

/// <summary>
/// Logs a message to the given log file
/// </summary>
/// <param name="logFile">The filename to log to</param>
/// <param name="text">The message to log</param>
static void WriteLog(string logFile, string text)
{
	//TODO: Format nicer
	StringBuilder message = new StringBuilder();
	message.AppendLine(DateTime.Now.ToString());
	message.AppendLine(text);
	message.AppendLine("=========================================");

	System.IO.File.AppendAllText(logFile, message.ToString());
}

This works great, but it still shows our user an unhandled exception message, even if we use the HandleError attribute. This makes the HandleError attribute look rather useless, so I've removed it. We can easily show the friendly error ourselves with the following code:

filterContext.ExceptionHandled = true;
this.View("Error").ExecuteResult(this.ControllerContext);

It's important to set ExceptionHandled to true, otherwise you'll still see the default unhandled exception message. The OnException method returns void so we must Execute the view and pass in the ControllerContext ourselves.

How Do I see my own Errors During Development?

It's a little inconvenient to open log files or keep commenting out your error handling code while developing to see exceptions and stack traces. You might remember ASP.NET has a nice web.config setting that configures custom errors. This property is exposed via MVC, so we can set up our config to show friendly errors to remote users only:

<customErrors mode="RemoteOnly" />

Then all we need to do in our OnException method is check this value and serve up the custom error view only if it returns true.

protected override void OnException(ExceptionContext filterContext)
{
	WriteLog(Settings.LogErrorFile, filterContext.Exception.ToString());

	// Output a nice error page
	if (filterContext.HttpContext.IsCustomErrorEnabled)
	{
		filterContext.ExceptionHandled = true;
		this.View("Error").ExecuteResult(this.ControllerContext);
	}
}

It's worth noting that IsCustomErrorEnabled will resolve the RemoteOnly option for you, you don't need to check where the user is coming from. Now out site serves up friendly errors to users and logs all exceptions without us losing the ability to see stack traces during development.

Using OpenID in your ASP.NET MVC Application/Blog

2009-04-14T19:23:00Z

Over the last few days I've been rewriting this blog in ASP.NET MVC. As it gets closer to a state where I can upload it, I found myself needing to implement security for the administration section (adding, editing posts, etc.). I don't want yet another username/password to remember, and I don't want to IP-restrict it because that's not very flexible (and I don't know how static my IP is!), so what are my options?

OpenID

OpenID is nothing new, it's been around since late 2005. I've been aware of what it did and how it worked, but never really played with it. I did, however, get the impression it might solve my problem. Especially having seen that you can use your Google account as an OpenID!

What's OpenID? Why is it Cool?

OpenID is a standard for authentication, allowing you to use the same identitiy/login for multiple services. It is not the same as using the same username/password at multiple websites (that's a very bad idea). Let's see an example.

I want to be able to login to my blog to edit posts. I don't want another username/password. As Google now works as an identity provider, my blog can redirect me to Google and let them authenticate me. Google will then return me to my blog saying "Yes, this is definitely Danny Tuppeny". This means I don't need any user tables, login forms, or anything else on my blog!

This might sounds complicated, but as with most things, there's a nice .NET library called dotnetopenid to hide the complexity. Let's see some code!

On the first request, dotnetopenid will return a null response. After logging in at the identity providers website, the user will be redirected back (to the same page by default, but this can be changed) with a token on the query string. This will cause dotnetopenid to return a response. The basic code looks like this:

var openId = new OpenIdRelyingParty();

if (openId.Response == null)
{
	// No response means this is the first page load
}
else
{
	// This means we're been redirected back after authentication
	if (openId.Response.Status == AuthenticationStatus.Authenticated)
		// User was logged in (as someone!)
}

On the first page load, we would usually ask the user for their OpenID Identifier/URL, however since in my case it's always going to be Google, I'm going to hard-code this as a single value.

dotnetopenid supports adding claim requests so that you can request (or even demand) specific pieces of information. In my case I only care about authenticating me, I don't need to request my name or email address. As such, I'm just going to fire a simple request off without any claim requests.

openId.CreateRequest("https://www.google.com/accounts/o8/id").RedirectToProvider();

In the else block we need to check the response. We want to make sure that the status is Authenticated and the ClaimedIdentifier matches the known identifier for my own login.

// We got a response - check it's valid
if (openId.Response.Status == AuthenticationStatus.Authenticated
	&& openId.Response.ClaimedIdentifier.ToString() == "http://google.com/blah/blah/blah")
{
	Session["Admin"] = true;
	return Redirect("/posts/edit");
}
else
	return Content("Go away, you're not me.");

The ClaimedIdentifier will be unique to each Google account. You can run the code once and examine the returned value to find out your own, and then you can check against it.

If we put all this together into a controller action, it'll look something like this:

public ActionResult Login()
{
	var openId = new OpenIdRelyingParty();

	// If we have no response, start
	if (openId.Response == null)
	{
		// Create a request and redirect the user
		openId.CreateRequest(Settings.AdminOpenIDIdentifier).RedirectToProvider();

		return null;
	}
	else
	{
		// We got a response - check it's valid and that it's me
		if (openId.Response.Status == AuthenticationStatus.Authenticated
			&& openId.Response.ClaimedIdentifier.ToString() == Settings.AdminClaimedIdentifier)
		{
			Session["Admin"] = true;
			return Redirect("/posts/edit");
		}
		else
			return Content("Go away, you're not me.");
	}
}

That's really all there is to it. Now when I hit the Login action I'll be redirected to Google's login page. After logging in, I end up back at /posts/edit on my blog with the correct session variable set. Of course, you could instead call the built-in ASP.NET authentication methods, or look up a user from your database based on their ClaimedIdentifier. There are a lot of ways you can extend this, and I'll cover using OpenID for blog comments in a future article!

IE8: Hanging with "Connecting..." when opening tabs, unable to hide Favourites bar and other bugs

2009-04-14T18:09:00Z

I'm not the only person having these problems, so I thought I'd post the solution here for all...

After hearing that IE8 will be offered via Windows Update next week, I decided to install it on my home PC running Windows Vista. I've been using it since it RTM'd at work with some major stability issues, but I put them down to my machine rather than IE. Oh, how I was wrong!

After the usual install and reboot cycle, I opened IE8. I turned off the usual trash (Accelerators, Web Slices, Compatibility View for Intranet sites, etc.) and went to hide the nasty favourites bar. Only, I couldn't. Right-clicking on the favourites bar didn't give a context menu. So I tried View » Toolbars » Favourites - the option was disabled!

I proceeded to Google, doing the usual middle-click on results to open them in new tabs. Every new tab I opened just sat with "Connecting..." in the tab title. The content was blank.

This isn't looking good...

I swiftly disabled all the non-MS addons (and Fiddler, thinking that could potentially break connections). No change.

I fired up Event Viewer and found an "Internet Explorer" event log. This would be an interesting find if the entire log wasn't blank. Great!

I opened up My Computer and navigated to my system drive. WTF - it opened in a new window. I checked folder options - it's still set to "open each folder in the same window". I don't like that :(

Rolling back to IE7 is now a serious option. This isn't what I've come to expect from Microsoft!

After a little more Googling (using Chrome, ofcourse) I found I wasn't the only one having these issues. Fixes varied from broken addons to GoogleUpdater. Nothing seemed to apply to my problem.

Then I found a magic post. Someoe had run IE as Administrator and discovered his problems disappeared. Even better, when he ran as a normal user afterwards, things still worked! Worth a shot, eh?

Well, it worked. Running as Administrator worked fine. Running again as a normal user and everything continued to work. Specifically, I could now:

Hide the IE8 favourites toolbar
Open new tabs without the "Connecting..." message/hanging
Open Windows explorer folders in the same window
Access toolbar context-menus

An interesting bug! I can't explain how it could happen, or why any errors/failures aren't written to the event log. All I can say is I'm glad it's fixed and I don't need to roll back to IE7!

Eager-Fetching of Relationships with LINQ to SQL

2009-04-13T11:34:00Z

By default, LINQ to SQL lazy-loads its relationships. This means it won't go fetching entire trees of objects when you're only using the top ones. However, this might not always be desirable. Imagine if every time you output a post for your blog, you include tagsa. You might write something like this:

foreach(var post in db.Posts)
{
	Response.Write("<h1>{0}</h1>", post.Title);
	foreach(var Tag in post.Tags)
	{
		Response.Write("<a href=\"{0}\">{1}</a>",
			Html.Encode(tag.FullUrl),
			Html.Encode(tag.Name)
		);
	}
}

This code will work fine, but if you were to examine the SQL being generated, you'd see n+1 SELECT queries. One for fetching the posts, and one for fetching the tags for each posts - individually.

What can we do about it?

LINQ to SQL allows us to specify a set of DataLoadOptions that dictate this behaviour. One of the methods of the DataLoadOptions is LoadWith<T> which allows us to say "whenever you load x, always include y". E.g.

// Create DataLoadOptions
DataLoadOptions dlo = new DataLoadOptions();

// Always fetch tags when we get posts
dlo.LoadWith<Post>(p => p.Tags);

// Set these options on the DataContext
db.LoadOptions = dlo;

If we re-run the original query, we'll now find a JOIN to the Tags table and just a single query to fetch all the data we require.

Simple! However, bear in mind that LINQ to SQL might not always generate joins. I've got a few cases where the LoadWith seems to be ignored. As soon as I figure out why, I'll be sure to update this post!

Reducing Duplicate Content with ASP.NET MVC

2009-04-09T18:54:00Z

As you're all no doubt aware, ASP.NET MVC recently went RTM. This brings the MVC-style of coding, made very popular by Ruby-on-Rails to the ASP.NET world. I've been eager to start using MVC for months, but I've been holding off until I knew the API was locked down so I don't have to change anything.

Unfortunately, like WebForms, MVC has some "issues" with regards to duplicate content, making it not all that SEO-friendly.

What do you mean, Duplicate Content?

Duplicate content is just that - the same content repeated on multiple pages/sites. This might not sound like a big deal, but it's not something search engines like. They don't want the search results to show the same content multiple times across different websites so they often penalise or hide duplicate content. Additionally, if you have two pages with the same content, your inbound links might become split between the two - reducing the pagerank passed to either.

What's this got to do with ASP.NET MVC?

Unfortunately ASP.NET MVC makes it easy to have the same content indexed multiple times. I've listed the main problems below.

Case-Sensitivity. In ASP.NET (or rather IIS and Windows), URLs are not case sensitive. That means you can write Default.asp, default.asp or even DeFalT.aSp and still get the same page. While you'll probably stick to the same case within your website, it wouldn't be hard for someone to create links to your site with different casing (e.g. they might have CAPS LOCK turned on).

Default Documents. Most websites have a default document set up to serve when a filename is not provided in the request. E.g. http://mydomain.com/ might actually serve up http://mydomain.com/default.asp, but it won't tell the browser that's what it did. It will serve it up as if the two are different URLs.

Trailing Slashes. While the above problems are general ASP.NET/IIS issues, trailing slashes are something that only really become a problem with MVC or other URL rewriting/routing. In ASP.NET if you requested http://mydomain.com/files and you had a folder named files, IIS would issue a redirect to mydomain.com/files/. However, in ASP.NET MVC the URL routing will treat trailing slashes the same as requests without. So http://mydomain.com/controller/action is exactly the same as http://mydomain.com/controller/action/ and therefore results in duplicate content.

Query Strings. Query strings can be a big problem for duplicate content. Imagine if you can add ?sort=field to the end of your page to have a table re-ordered. To a search engine this looks like another page, but the content is mostly the same. Fortunately, ASP.NET MVC doesn't really use query strings thanks to the excellent URL routing.

So, what can we do?

Lowercase URLs. We can force all requests to our application to be lowercase by catching them in BeginRequest in Global.asax and redirecting to the lowercase version if they contain any uppercase characters.

protected void Application_BeginRequest(Object sender, EventArgs e)
{
	// Get the requested URL so we can do some validation on it.
	// We exclude the query string, and add that later, so it's not included
	// in the validation
	string url = (Request.Url.Scheme + "://" + HttpContext.Current.Request.Url.Authority + HttpContext.Current.Request.Url.AbsolutePath);

	// If we've got uppercase characters, fix
	if (Regex.IsMatch(url, @"[A-Z]"))
		PermanentRedirect(url.ToLower() + HttpContext.Current.Request.Url.Query);
}

/// <summary>
/// Redirects with a 301 header to pass along any incoming
/// PageRank/link value.
/// </summary>
/// <param name="url">The URL to redirect to</param>
private void PermanentRedirect(string url)
{
	Response.Clear();
	Response.Status = "301 Moved Permanently";
	Response.AddHeader("Location", url);
	Response.End();
}

Now if anyone requests a URL with uppercase characters, they'll be redirected with a 301 redirect. This works great, but we have a problem. All URLs generated internally by MVC will continue to use Action and Controller names in Pascal case (assuming that's how your classes are named). This means every link within our site will cause two requests (the first being a redirect). To fix this, we can override the default behaviour for creating URLs. We'll create a new extension method for the RouteCollection class called MapRouteLowercase which instead of creating a Route will create an instance of a new class, called LowercaseRoute. This class will override the GetVirtualPath method to lowercase the URL before passing it back. I can't take credit for this code, I pretty much just copied it from Graham O'Neale's blog.

public class LowercaseRoute : System.Web.Routing.Route
{
	public LowercaseRoute(string url, IRouteHandler routeHandler)
		: base(url, routeHandler) { }
	public LowercaseRoute(string url, RouteValueDictionary defaults, IRouteHandler routeHandler)
		: base(url, defaults, routeHandler) { }
	public LowercaseRoute(string url, RouteValueDictionary defaults, RouteValueDictionary constraints, IRouteHandler routeHandler)
		: base(url, defaults, constraints, routeHandler) { }
	public LowercaseRoute(string url, RouteValueDictionary defaults, RouteValueDictionary constraints, RouteValueDictionary dataTokens, IRouteHandler routeHandler)
		: base(url, defaults, constraints, dataTokens, routeHandler) { }

	public override VirtualPathData GetVirtualPath(RequestContext requestContext, RouteValueDictionary values)
	{
		VirtualPathData path = base.GetVirtualPath(requestContext, values);

		if (path != null)
			path.VirtualPath = path.VirtualPath.ToLowerInvariant();

		return path;
	}
}

public static class RouteCollectionExtensions
{
	public static void MapRouteLowercase(this RouteCollection routes, string name, string url, object defaults)
	{
		routes.MapRouteLowercase(name, url, defaults, null);
	}

	public static void MapRouteLowercase(this RouteCollection routes, string name, string url, object defaults, object constraints)
	{
		if (routes == null)
			throw new ArgumentNullException("routes");

		if (url == null)
			throw new ArgumentNullException("url");

		var route = new LowercaseRoute(url, new MvcRouteHandler())
		{
			Defaults = new RouteValueDictionary(defaults),
			Constraints = new RouteValueDictionary(constraints)
		};

		if (String.IsNullOrEmpty(name))
			routes.Add(route);
		else
		routes.Add(name, route);
	}
}

You can put these classes anywhere. Because MapRouteLowercase is an extension method, you can just call it on the RouteCollection class in place of the existing MapRoute call in your Global.asax.

// Home stuff
routes.MapRouteLowercase(
	"Default",
	"{page}",
	new { controller = "Home", action = "Index", page = 1 },
	new { page = @"\d+" }
);

Default Documents. While this issue doesn't affect MVC in the same way, there's a very similar problem. In ASP.NET MVC the default routing is {controller}/{action} but it sets a default action of Index. That means on a newly-created project, both /Home/Index and /Home will serve up the same content.

To work around this, and provide some nicer URLs, I changed the routing a little so that my default actions where mapped to the root and a seperate route dealt with the homepage (which accepts pages, to allow browsing to older posts).

public static void RegisterRoutes(RouteCollection routes)
{
	routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

	// Posts
	routes.MapRouteLowercase(
		"Posts",
		"posts/{url}",
		new { controller = "Post", action = "Display" }
	);

	// Tags
	routes.MapRouteLowercase(
		"Tags",
		"tags/{url}/{page}",
		new { controller = "Tag", action = "Display", page = 1 },
		new { page = @"\d+" }
	);

	// Home stuff
	routes.MapRouteLowercase(
		"Default",
		"{page}",
		new { controller = "Home", action = "Index", page = 1 },
		new { page = @"\d+" }
	);

	// Home stuff
	routes.MapRouteLowercase(
		"Home",
		"{action}",
		new { controller = "Home", action = "" }
	);

	// Catch-all for any unmatched URL
	routes.MapRouteLowercase(
		"Error Catch-All",
		"{*path}",
		new { controller = "Home", action = "NotFound" } // NotFound doesn't exist, so HandleUnknownAction will be fired
	);
}

Trailing Slashes. To avoid trailing slashes and a few other minor issues (such as people adding /1 to a URL to get page 1, which is served up without the /1) I added some additional rules to my Global.asax as below.

protected void Application_BeginRequest(Object sender, EventArgs e)
{
	// Get the requested URL so we can do some validation on it.
	// We exclude the query string, and add that later, so it's not included
	// in the validation
	string url = (Request.Url.Scheme + "://" + HttpContext.Current.Request.Url.Authority + HttpContext.Current.Request.Url.AbsolutePath);

	// If we're not a request for the root, and end with a slash, strip it off
	if (HttpContext.Current.Request.Url.AbsolutePath != "/" && HttpContext.Current.Request.Url.AbsolutePath.EndsWith("/"))
		PermanentRedirect(url.Substring(0, url.Length - 1) + HttpContext.Current.Request.Url.Query);

	// If we end with /1 we're a page 1, and don't need (shouldn't have) the page number
	if (HttpContext.Current.Request.Url.AbsolutePath.EndsWith("/1"))
		PermanentRedirect(url.Substring(0, url.Length - 2) + HttpContext.Current.Request.Url.Query);

	// If we have double-slashes, strip them out
	else if (HttpContext.Current.Request.Url.AbsolutePath.Contains("//"))
		PermanentRedirect(url.Replace("//", "/") + HttpContext.Current.Request.Url.Query);

	// If we've got uppercase characters, fix
	else if (Regex.IsMatch(url, @"[A-Z]"))
		PermanentRedirect(url.ToLower() + HttpContext.Current.Request.Url.Query);
}

This seems to stop many of the issues I came up with, however the double-slash seems to be passed through (in AbsolutePath) as a single slash here (Vista/IIS7) so doesn't work. I've left it in just in case this behaves differently on other web servers.

Is there anything else I should do?

As of February, Google, Yahoo, ASK and Microsoft Live Search support a new Canonical meta-tag. This allows you to specify on a page that this page is duplicate content and any incoming links should instead be attributed to another page. If your site has query strings or other potential for multiple requests to serve up the same content I would recommend inserting this tag to make sure the search engines choose your prefered page.