Entries for label: amazon

Found 6 entries.

Playing with EC2

Once upon a time, I worked on the European Space Agency's Beagle2 probe which was supposed to land on Mars. Well, this isn't a story about that. That would be too sad.

But I do remember telling my brother that for the first few weeks of the project, I was "playing" with the prototype board trying to get it to run some Ada 83 programs. He started laughing at me and I asked why.

"Playing with something that costs millions of pounds - doesn't sound right to me."

But I explained to him that the only way to learn something is to have a play with it. That's what kids do and sometimes as adults we forget that it's not just one way to learn but the best way to learn.

Here's an opportunity to find out what all the fuss is about over the Amazon Web Services, specifically EC2. I have written an article called Getting Started with EC2 so just go over there and have a play.

And the funny thing is, it's just so easy, it's almost childsplay.

Labels: project-awssum, planet-geek, ec2, planet-catalyst, amazon

Inserted: 2008-04-03 23:45 (3 years, 10 months ago)

Introducing Project AwsSum

There's a lot of excitement happening in 'cloud' computing at the moment and I'm not surprised. People are finally realising the usefulness of fully distributed and tolerant systems.

I have harped on about Amazon and their Web Services on a number of occassions (dating back to October 2006).

Back when I first wrote s3bak I downloaded and installed the example Perl code Amazon provided 'as is'. But that wasn't good enough, I wanted to know more about the protocol itself.

Of course, there's no better way of learning something than actually doing it so I started implementing a library to talk to SimpleDB. Then I figured out that SQS and EC2 were pretty similar beasts and it just so happens that S3 could also be done in a similar way.

After a few weeks of hacking, I now have a reasonably complete set of simple modules which can talk to all of SQS, EC2, S3 and SimpleDB.

Project AwsSum comes with some straightforward command line scripts which allow you to execute commands against the services and see what's actually being queried under the hood.

However, the most interesting thing I've been working on are the new types of programs I would never have written if these services didn't exist.

As an example. I did 2 lightning Perl Mongers talks in March. The first was Talking to Amazon Web Services and the second Playing with Amazon Web Services (as always in lightning talks, they weren't long enough or in-depth enough). But I did write a program to demonstrate the power.

Using both SQS and S3, I wrote 5 simple programs to interact using only the queue as the interface between them (and only 3 of these were necessary). It's a simple demonstration but it shows the power gained by writing loosly coupled code. Quickly running through them:

  1. put-image.pl - just uploads images to S3 and places the filename in the queue
  2. process-images.pl - polls the queue for messages and if it finds one, downloads the image and creates thumbnails of different sizes. It then re-uploads these new images into S3.
  3. generate-gallery.pl - generates and index HTML file using a list of what's in S3

(report-queue-progress.pl and report-bucket-contents.pl just report on the status of the queue and the files being stored.)

So in about 350 lines of code (130 being the processor) I have a simple gallery which uses no webserver, is fully distributed, is resilient to hardware failures and was written in less than a couple of hours. It also comes with an Amazon SLA which is higher than you could do yourself. And did I tell you the price - all this for around 3 or 4 US cents per month. Not bad eh?

If you want to have a play with Project AwsSum you can git clone at git://github.com/andychilton/awssum.git. I'd love to hear feedback - and remember, it's a work in progress :-)

P.S. I also have a tutorial giving examples of how to get started with EC2 using some of the newer features which have only just been released! Seeing as Geoff is giving a talk to us at Pizza Thursday tomorrow, I don't want to give the game away (just for you vex).

Update: so Pizza Thursday is *NEXT* week - I got it wrong. I apologise. Pizza Rage ensues on #catalyst.

Labels: project-awssum, planet-geek, planet-catalyst, amazon

Inserted: 2008-04-02 20:12 (3 years, 10 months ago)

Amazon Web Services with Mike Culver

Tonight, we got a good opportunity to hear and see what Mike Culver (Web Services Evangelist) had to say about Amazon Web Services.

It was a good talk. A bit fast to go through everything but that's a reflection of how much Amazon have to offer. Lucky for me, I knew most of it but I did learn a few things along the way. Especially from the demo of using EC2. I wish he could have spoken more about SimpleDB though.

As it turns out, my plan to ask if I can be added to the Beta program for SimpleDB worked and I have just sent an email off asking if I can join it. So yeah, that makes me very happy.

The good thing is, as you know, I've been playing with S3 and s3bak a lot recently, but I've also been making a program to play with EC2 as well. It's actually not that big but already there is some good functionality. My plan to make a small command line interface to both SimpleDB and SQS is also in the pipeline.

It certainly is exciting to be playing with these technologies and while there are some criticisms against them, I'd say to see them as they are and watch them advance over time. The whole point of the whole setup is to be easy to use and it's up to us application developers to use them in new and enlightened ways, adding value along the way.

However, I do see a different problem though - oh so many ideas and oh so little time.

2008-02-20 10:31 - Update: I've just been added to SimpleDB Beta. Wow, they work fast :-)

2008-02-20 19:32 - Update: Don Christie (president of NZOSS) has linked to this article from his Pass the Source blog - When Amazon Comes to Town. Out of the 4 quotes of feedback, mine was the top one and one which I will try and expand into a bigger article later.

Labels: ec2, simpledb, aws, mike-culver, sqs, planet-geek, planet-catalyst, s3, s3bak, amazon

Inserted: 2008-02-19 22:07 (3 years, 11 months ago)

Some Cool Stuff Coming Up

There are a few things in February I'm really looking forward to.

Firstly, it looks like Grant has snared Damian Conway to come and speak to us at Wellington.pm . That is going to one awesome evenings.

And secondly something which made me smile. I read the Amazon Web Services blog and saw this article about New Zealand and Australia the other day. So I followed their wiki entry to this page about Mike Culver's Schedule only to realise that he's already coming to WellyLug/Catalyst :-) It seems that Brenda was to blame for this - so thanks Brenda.

Also, Brenda, Reed and co. have been busy figuring out which language to use next (and it even got posted on PerlBuzz). I'd have also said Ada is Anal but then, I had to do Ada 83 (Ravenscar Profile) for 9 months once.

Labels: programming, planet-geek, aws, perlmongers, amazon

Inserted: 2008-01-10 00:28 (4 years, 1 month ago)

S3 Revival

After lately talking to different people about the Amazon Web Services, I have become interested in them again.

And after ages and ages wanting a place 'in the cloud' to store my files, I decided that I'm not going to wait for Google's GDrive, but instead am going to use Amazon's S3.

So, all those old photos I have dotted around are currently being uploaded. I'm using a program I wrote, s3bak, and whilst I haven't touched it much in a year, it's still working fine. It's even got some nice features. Think I'll be hacking a little on that in the next week or so.

Anyway, it'll be more important to me tomorrow when I finally get myself a new digital camera. It's not top of the range but the price/functionality is exactly what I wanted. The Canon A570 IS isn't too expensive but has some of those things I like, such as aperture and shutter priority.

I also think things like S3 are underused in some commercial senses. There's certainly one thing I'd love to use it for in work and it would save buying a lot of hardware and save a lot of hassle.

As I said to someone recently, "because it's on the same system Amazon use, if something goes wrong, they would stand to lose a lot more money than you would" - hence, the reliability of the thing is phenomenal. If it's good enough for Amazon, then it's good enough for my backups.

I also intend to use it for various web projects I have brewing too. Not to mention EC2 and the other Amazon Services.

Labels: photos, planet-geek, ec2, planet-catalyst, s3bak, s3, amazon

Inserted: 2008-01-06 23:36 (4 years, 1 month ago)

s3bak Out Now

After playing with Amazon S3, I decided to write a little tool to help.

At the moment it is a first cut with no Debian packaging, but so far it has a nice amount of functionality.

You can create and remove buckets, list keys, add files, delete files but the best part is the ability to upload all missing files (in S3) and download all locally missing files.

At the moment, it doesn't do anything if the key exists and is the same name as a local file, but I did have some code to check the MD5 and upload it if it has changed. This gives the problem that you might have changed the S3 files from somewhere else, therefore I will write a 'diff' which will have to be used manually to upload/download the correct version of the file.

Oopse, I'm getting technical. Anyway, for now, it seems to work and does most of what I want. Go take a look at: s3bak at Google Code.

Labels: s3bak, s3, amazon

Inserted: 2006-10-22 23:43 (5 years, 3 months ago)