The web testing puzzle is a jigsaw with 3 pieces, and for a long time the 3rd piece has been hiding down the side of the sofa, but I think Simon Stewart just found it. He's called it WebDriver - it needs dusting off, but it's looking like it's going to be the right fit.

The problem with testing websites is that it has always been messy. It's bound to be hard coordinating 3 tasks (browser, server, set of tests) across at least 2 processes on different hardware setups, and maybe even different platform architectures.

The set-of-tests piece of the puzzle is well understood; JUnit, TestNG, blah blah.

The well behaved server piece is solved for Java by Jetty - you can fire up an embedded server as easy as new Server(); (example).

Now we can make browsers be as well behaved - we can remote control web browsers from within a Java program. WebDriver currently supports Internet Explorer, Firefox and HtmlUnit for totally embedded work.

WebDriver driver = new FirefoxDriver();
driver.get("http://localhost:8080/dwr/simpletext/index.html");

driver.selectElement("id=demoName").setValue("World");
driver.selectElement("id=demoSend").click();

// Give the browser a chance to do the onclick
Thread.sleep(500);

WebElement reply = driver.selectElement("id=demoReply");
assertEquals("Hello, World", reply.getText());

Scott McMaster has some more examples.

Just to be clear HtmlUnit has existed for a while, and it's great for testing the sunny day case where browsers work according to a spec. It's complete enough to be able to run DWR for example, but it won't help you test the weird corner cases. WebDriver helps by reusing the weirdness directly from the browsers.

Update: Patrick Lightbody points out that Selenium Remote Control does something similar. The core Selenium controller class actually looks to be more capable than the WebDriver version, I try it out to see if it embeds as well.

A couple of weeks ago I recorded a Webinar on DWR (see previous post), which TIBCO are kindly hosting as a flash movie. It contains an overview of Reverse Ajax, integration with the Open Ajax Hub and TIBCO GI, and a new call center demo that will be part of DWR 2.1.

Sys-Con have published an article: "Simple Streaming AJAX Comet App with Open Ajax Hub, TIBCO GI, and DWR", that I wrote with Kevin Hakman. The main article link is here or you could try the print version.

Philip McCarthy has written an excellent article on "Writing scalable Comet applications with Jetty and DWR":

The Comet pattern allows you to push data to clients, and Jetty 6's Continuations API lets your Comet application scale to a large number of clients. You can conveniently take advantage of both Comet and Continuations with the Reverse Ajax technology in Direct Web Remoting 2.

If you are interested in how Jetty saves threads under heavy Comet load, it's well worth a read.

Conferences

I don't normally blog every time I go to a conference because I'm sure I would sometimes forget, and it's probably not that interesting (or is it? argue in the comments), however there are a few that are coming up:

NFJS Exchange: London / August 29-31

No-Fluff-Just-Stuffs are cool, and I particularly like that they are coming to the UK after being a US only event for years. What's good is the size - only 250 attendees, so it's way more intimate than JavaOne or something similar.

Plus there's always good swag at NFJS conferences, the swag for TRWE (below) is well known, but I'm told there's a free Wii to all attendees just for quoting a code: NFJS-DWR667. Register here.

I'm talking on DWR and on Ajax Security. I may also be sharing some stage time with Mark Goodwin.

The Rich Web Experience: San Jose / September 6-8

TRWE has an awesome list of speakers - I am looking forward to hearing Aza Raskin again. He was great at the Ajax Experience last year. I've not seen any conference that lets attendees choose from a Wii or a 30Gb iPod Video before, plus if you book using the nfjs2007speaker200 promo code you can get $200 off the entrance price. Register here.

I'm talking on DWR and on Ajax Security.

Future of Web Apps: London / October 3-5

FoWA is very different for me because it's not a full time developers conference. They cater for developers, but also to a range of other skills, which means there are a whole bunch of interesting speakers that I've not met before. I'm talking on Comet and DWR. Also interesting is their web-app for registered attendees, so you can see who else is going (obviously, you need to register to attend to see), and free beer road trip.

Grails Exchange: London / October 17-19

The Grails Exchange is a little strange because although there's there's a lot of Grails actions, there's also quite a lot of non-Grails-centric stuff going on too. I'm talking on DWR and on Ajax Security. If you're quick there's still early-bird registration.

The Ajax Experience: Boston / October 24-26

TAE / Boston is a way off; I've only just come home from TAE in San Francisco, so I'm way ahead of myself. We've not even arranged subjects yet.

Web security is horribly broken, and lot has been said about CSRF, XSS, DNS-Pinning, etc, but not enough about what we can do to fix the mess.

I think we could adapt an idea like HttpOnly to tackle CSRF - I'd like to see a "SameRefererOnly" marker for cookies.

SameRefererOnly is an indication that a cookie should only be sent to a Site when the referring domain matches the destination domain.

A number of people have commented that you could use server based referer checking to fix CSRF, however that doesn't work for 2 reasons, firstly sometimes referers are not sent, and secondly using old versions of Flash, you can forge referer headers anyway.

However if we move the checking into the browser, then we should be able to instruct browsers to be more careful what they do without our cookies.

The official spec for Set-Cookie is RFC 2109/2965 which says this:

Set-Cookie: <name>=<value>[; <name>=<value>][; expires=<date>] \
            [; domain=<domain_name>][; path=<some_path>][; secure]

Microsoft's HttpOnly spec adds this to the end:

            [; HttpOnly]

I propose that we further add this:

            [; SameRefererOnly]

Any cookie marked SameRefererOnly MUST only be sent to servers when the domain of the cookie matches the referring domain. The cookie MUST NOT be sent when there is no referring domain (this might help to prevent CSRF / Phishing mash-ups).

It's worth noting that this protection breaks in the presence of XSS flaws in a site, because XSSed commands come from the compromised domain, however that does not make this protection useless.

Plug-ins like Java and Flash that have their own HTTP pipelines generally don't send cookies so these systems probably do not need to be patched.

This proposal has 3 places where it can come in handy:

• As an additional defence-in-depth layer of protection for a site that already has other CSRF protection.
• On an Intranet site where access through browsers that implement SameRefererOnly can be mandated
• In a future world where enough browsers implement this, that other browsers can be excluded.

Note on spelling: The pedant might note that this should be spelt SameReferrerOnly - i.e. with a double 'r', and the pedant would normally be correct however this misspelling is so common that the HTTP spec makes it, and hence we all follow unquestioningly.

I'm sure there must be some holes in this suggestion, but I make it anyway so someone can move the discussion on.