How Mechanical Sympathy got me to the airport on time

Lets talk about mechanical sympathy.  Martin Thompson has been making this term very popular in software development, so it's best to read his description of why he used the term.

I had a perfect example yesterday.  I'm about to drive to the airport and the car won't move (I'm a modern tomboy, I can write stories about hair and stories about cars). Not at all.  It's stuck.  I can't reverse out of my parking space.

The first thing that occurs to me is something is stuck under the car.  Like a cat.  Those buggers hide in the stupidest of places.  So I get out and check there's nothing wedged against the wheels, which seems like the most logical thing that would stop the car moving.

(OK that's not true.  The first thing that occurs to me is oh-my-god-I'm-going-to-miss-my-plane-and-I-haven't-got-a-backup-plan-to-get-to-the-airport-and-I've-already-paid-for-parking-and-I-would-like-to-cry-now-but-that's-not-going-to-help)

Since there's nothing under the car, the next thing that occurs to me is the handbrake is stuck on - it feels like it's the rear left wheel that won't move, and I know (probably as I had a similar problem on my previous car, my lovely but ancient MX5, or maybe because I've seen far too much Top Gear) that the handbrake applies to the rear wheels.

I also suspect that the handbrake jamming this is the correct answer because I put the car through the car wash on Monday, and the known issue of the brakes jamming on some Volkswagens always manifests in my car after the car wash.  I should probably just let it get dirty.

Because I'm big on tests, and scientific theory, I test this hypothesis.  I rev the hell out of the car and force it backwards.  OK, I'll be honest - this wasn't a test.  This was a brute force attempt to unjam the handbrake. But it did provide a suitable test - when I got out of the car to inspect the offending wheel, I could see skid marks on the floor where the wheel had been dragged, rather than rolled, over the tarmac.  Oops.

So my (limited) knowledge of how a car works has provided me with a hypothesis, and reasonable proof that it's correct.

That's all fine, but I'm still not on the way to the airport yet.  How did I fix it?

What do you think I did? Well, I drove it back and forth a view times, hoping the jolting and extreme discomfort of the brake pad against a moving wheel would unlock the brake (from previous experience, the brake clicks off, rather than simply easing off).

Of course it didn't work, and I'm probably going to have to replace that tyre much sooner than expected.  Ooops x2.

I sat in the car and pumped the foot brake millions of times - a trick that either inexplicably worked on the MX5, or simply distracted me long enough and the brakes fixed themselves eventually.

This didn't work, not surprisingly (not the same brakes after all).

Finally, I got out of the car and kicked the wheel a few times.  Hard.  I figured this might jolt the brake back into position.

It didn't

Then I used my brain.  That's not going to work when you unconsciously put the hand brake on when you leave the car to kick the wheel.

So I took the brake off, then kicked the wheel.  Hard.  A lot.

And it worked!!!

And this is a bit like diagnosing performance problems (no, really).  My basic understanding of how the car was put together, plus some experience with similar problems in the past, gave me guidance on where the issue was and what might be done to fix it.

So, what have we learnt today, boys and girls?

• A basic understanding of how the hardware works can prevent you calling an expensive expert to do a simple fix.
• Always have a backup plan (Disaster Recovery/High Availability - if not hardware, then at least some sort of process).  I was much more stressed because without the car, I had to come up with a whole new transportation plan with a very limited time budget and having no knowledge of or experience with alternatives.
• I need to drive my car more, because then a) I would have either discovered the problem sooner, or b) it would never have got to that point.  I'm going to say that in this tenuous analogy, the equivalent is to do testing in a live-like environment, and actually do DR scenario testing.  Then you know how your hardware and software behaves under abnormal circumstances, and have some concept of how to get back to normal.

I did make it to the airport on time, because adding padding to your estimates is a Good Thing and gives you a bit of space for contingencies.  So this story does have a happy ending.

Here endeth the lesson.

Devoxx UK 2013

Last week was the first Devoxx UK, bringing the brand from Belgium and, more recently, France.  And I think it was a HUGE success.

Of course, disclaimers first - I was on the programme committee, so I might say that the whole thing was awesome.  Although in all honesty, even being slightly involved in the organisation of a conference is a lot of work, and you can't wait to see the back of it (and I didn't do half as much as some people).  Note to self - never speak at a conference you're on the programme committee for, it's too much work.

As a speaker and attendee though, Devoxx was really brilliant - great speakers, interesting content, awesome friendly atmosphere, good venue, parties with free drink and good opportunities to speak to people.  There were downsides: limited coffee availability, no free croissants, um... I'm really struggling to think of anything else, and none of those are show-stoppers (especially when the venue is in Islington, so you can step outside and get whatever you want).

All throughout the run up to the conference we kept hearing it was a conference "for developers, by developers" until I was nearly sick of it.  But it was a mantra that kept us on track when we were deciding on content, and it really came through on the two days of the conference.  Pretty much everyone on stage was, or had been, a developer, the talks were a good balance between "useful" stuff for our day jobs and "interesting" stuff for techies <insert joke about some being "useful" and "interesting">.  The future track excited us with things like like hacking on the Raspberry Pi, and the rest of the conference had a good mix of Java SE, EE, methodology and Other Stuff.

For me, the highlights were:

• Kevlin Henney's keynote, reminding us that Developers Are Human.  Who knew?
• Sandro Mancuso's walkthrough of OO principals, and how they apply to us in our actual day jobs.  Sandro did a good job of taking quite a dry topic, that many of us dozed through at university, and showing us why we already use these principals and how understanding them will make us better programmers.
• Mazz Mosley had an entertaining talk about the Dark Side of Agile, walking us through the agile practices in The Government and highlighting many of the unanswered questions that teams have after switching to agile.
• Colin Vipurs's talked about Test Driven Development from the trenches - he managed to give a good intro for those who might not have done it in practice, but for those who may have been doing it for a while, this a) was a good reminder of good practice and b) gave a good insight into why we do it.  On top of that, he had solid advice on how to write better tests, that we could all take home with us.
• Martin Thompson gave the talk I was attempting to give at JAX London last year - what is performance and how do we achieve high performance in Java?  Where I failed, Martin really rocked it.  I'll be leaving that to the professionals.
• It would be rude of me not to mention this point - what a diverse conference!  We had developers of all shapes and sizes, from the suits to the mohawks, of all colours, and (by technical conference standards) lots of women.  Watch the video below and I think you can see it.  I was so proud to have been a part of it.
• ...and meeting everyone.  What an awesome conference for networking!  Of course, there was a large number of London Java Community people there (speaking, volunteering, and attending), and quite a few familiar faces from Devoxx Antwerp - it was brilliant to see them in London.  But I was also very pleased to run into people I've worked with in the past, some I haven't see for ages, and some I still see regularly.  And I met dozens of new, interesting people - you guys really made the conference for me.

I was giving my new presentation, What Do You Mean, Backwards Compatibility?  In this talk I share some of the experiences I've had in helping to develop the new MongoDB Java driver.  This is not really a MongoDB talk, although there are some examples from the new driver, it's more a talk about design, and how we as developers make design decisions every day, something I don't think we recognise very much.  As always, I'll post a link to the video when it becomes available.  However, I am hoping to give this presentation a number of times this year, so if you missed it there will be another opportunity to see it.

As usual, my take-away theme from a conference is probably influenced by the sessions I attended, so is as much a function of what I think is important right now as it is an indication of what themes are emerging in the industry.  But I came away from Devoxx feeling like there's a desire to focus on the basics - the OO principals that have been around for years; the good design practices that have been written about many times; and, most of all, a reminder that developers have a difficult job - we don't simply turn requirements into code, we are constantly doing tricky stuff like system design, and even trickier stuff like working with other people.

I loved Devoxx UK, and I wasn't alone in feeling a bit empty when it was all over.  To re-live a tiny amount of the event, or get a good feel of what it was like, watch this video by the awesome Roy van Rijn.

Devoxx UK 2013 - Long mellow version from Roy van Rijn on Vimeo.

As with all Devoxx events, the talks will be available on Parleys.  If you watch nothing else, watch Kevlin's keynote.

It Depends

Don't you hate it when you ask a perfectly good question, and someone comes back with the answer "it depends"?

It's so frustrating to think that in a world of ones and zeros, people can't give absolute answers and you can't rely on "best practice".

It's an answer I've given so many times, especially when someone asks about performance.  Well, I've had my comeuppance.  The entire exercise of designing the new Java driver for MongoDB has been nothing but a series of questions where the answer is "it depends":

• Which Java version are our users, um, using?
• Do people want an asynchronous driver?
• How will they want to work with async?
• Will they want to use async and synchronous method calls from the same application?
• Do people typically use the Java driver directly, or do they use something that wraps it, like Morphia or Spring Data?
• What's most important for users in terms of performance?  Throughput? Latency? Consistent GC profile? Something else?
• What sorts of operations are our users doing?
• Do people usually update their driver and the server version at the same time?
• Is it easier for them to update their driver version than their server version?
• Do they use (or will they need) custom encoders and decoders?
• When should we deprecate?  When should we remove deprecated methods?

...and so on, and so forth.

It's such a change from the sort of development I'm used to: "the business" (a business analyst, a customer, a business owner) comes to you with a requirement, you ask a bunch of questions, preferably explaining the trade-offs that come with decisions or approaches, and then you and the team come up with a design and implement it.  If you're agile, this is all done in a nice, iterative fashion, which hopefully leads to "the business" being happy, or at the very least to another series of stories/requirements to work on.

The problem is that working on a library, particularly an open source library, is a completely different thing.  We don't even know who our users are.  This statement is true for pretty much any web application, of course, but there at least you can (if you choose) use tools like analytics and A/B testing to figure out what works for your users and what doesn't.

When your library is used for free by all kinds of different teams and companies, including organisations behind closed doors, you have no idea what's being used, what works, what people like, what people don't like.  The most visible feedback you have is when you see blog posts telling everyone how bad your product is.

This makes the design exercise VERY difficult.  Take performance for example.  Having come from a high performance, low latency background, I'm desperate to have a very extensive suite of automated performance tests (and we do already have some).  But how do I design those, when I don't know:

• What operations are typical for our users
• Whether our users care more about latency or throughput, or mean latency vs the long tail, or GC pauses, etc
• How much data customers tend to punt around
• What the hardware or network topology looks like?

The number one lesson you learn when performance testing is to make your test environment as similar to production as possible.  How can we do that for all our customers?

Well, we can't.  Of course.

What we can do is offer an easy way for our users to test it for themselves, with their own data, their own hardware, their own use-cases.  If we can provide some sort of hook into standard metrics, we could get users to plug into that and do what they needed.  In theory, as we get more examples of these standard metrics from a range of users, it will be easier for us to help them diagnose problems.

What I discovered thinking about this problem is that I was asking the wrong question - it's not "how do we test this?" but "how do we make it as easy as possible for users to test this in a production-like environment?".

My biggest headache has been backwards compatibility.  I've worked on plenty of systems where we've provided an API which has to be maintained in a friendly way for those who use it.  That's tough enough - you have to be careful to only add methods, not to change signatures or remove them altogether.  But when your system is not simply an API to code against, but a library that runs within other people's systems, this problem is even harder.  Greg Young talked at QCon about this problem from the developer's point of view - every piece of code in your system, even if it's a third party library, is your code.  Because you're the one who'll get called at three in the morning if your system falls over with a ConcurrentModificationException in some third party data structure.

So as library developers, we have to not only provide excellent quality, well tested code, but we also have to let our baby go off and run in strange environments.  Ones that might not be running Oracle's Java 1.7, ones that might contain other libraries that could clash with our own.  Have you ever looked at all the Maven dependencies in a large project?  You can end up with conflicting versions of common libraries (e.g. logging or DI frameworks) as every library you use pulls in dozens of libraries for itself.

In our case then, we need to use as few dependencies as possible, and to write nice, clean, modern Java, whilst supporting older versions of Java.  What's modern enough?  We know that some large organisations take a loooong time to upgrade to the latest versions of Java.  We currently support Java 5 and above, as 5 was a big enough change (and is old enough now) to be a good point to draw the line.  But what about Java 7, with its shiny new asynchronous channels?  That would be awesome for a driver like ours, an application that exists solely to connect to some server somewhere.  What about Java 8, with lambda goodness and a very appealing Stream interface, that supports (and encourages) a syntax that looks like it might work well for providing a fluent API for querying, I dunno, databases?  How do we make the most out of advances in modern Java, without alienating our existing users?

And, of course, I haven't even talked about actually changing the API.  The existing Java driver doesn't even make use of generics.  How can we change our API to provide a more modern-looking interface, without either forcing all our users to make massive changes to their applications (and for what?  Their code already works), or adding so many new classes and methods that it becomes very difficult for new users to work out the Best Practice way of interacting with our driver?

So, what can we do?

• Firstly we have to figure out which questions we have and what possible solutions exist.
• Then we have to weigh up the pros and cons of each of the possible solutions.
• Ideally we'd get feedback early and often from our users, from the community, about the approaches we're taking.
• Development would happen in parallel, in a nice, agile, way, to bring the best possible solution for everyone.

What this all really means, though, is that the Shiny New Java Driver™ is not ready right now, and will not be ready immediately, despite the fact that we've already spent some time on its development, and considered all of those questions and more.  Right now, we're starting to get feedback from the community - from users, and from committers (or people who would like to be committers).  Which means that I get to to go to more conferences and user groups, and talk about our problems when designing the new driver.  I'm hoping to get two things out of this, apart from more air miles:

1. Feedback from the community about our assumptions and the direction we're taking.
2. Present to the development community some of the lessons we've learnt/are learning, in the hope that you can use them when approaching the design of your own applications.

<gratuitous-conference-plug>You've already seen some of these upcoming events.  But in case you haven't, this Thursday I'll be presenting at Skillsmatter on how we approached this design problem, and again at DevoxxUK next week.

If you're more interested in everything MongoDB, and want to get a much better look at what it is, how it works, how to design for it, then come to MongoDB London, where I am one of numerous presenters, all going into MongoDB specifics.</gratuitous-conference-plug>

So... I'm afraid to say that even after this long post, even after bemoaning my own experiences of hearing "it depends", I still don't have an answer for you.

But maybe I do.  

"It Depends" means "you need to get more information in order to answer that question".  And it's our job as developers to ask the right questions to gather that information.  If the answer is "it depends", you're not asking the right question yet, or you're not asking the right people.  So dig down, find out why you can't answer the original question, and start iterating through your design process until you have answers that you can act upon.

Easy.  Right...?

Upcoming Events

Following on from a busy and fun QCon, I can now let you in on the next set of events I'll be at:

• 21st March - LJC: What do you mean, backwards compatibility?
• 27th March - DevoxxUK: What do you mean, backwards compatibility?
• 27th March (evening) - Hope to attend the London CTOs meetup.
• 9th April - MongoDB London: Schema Design
• 10th April - East Anglia MUG: Trisha Gee Unconference
• 15-17th May - GeeCON, Krakow
• 18-19th June - GOTO Amsterdam (I think, although I'm not on their site!)

Note that I'm actually going to be in the UK for all of March and April (at the moment), which will be a nice change.

I'm hoping to be able to speak about the same types of things for the rest of the year, it has been exhausting presenting on new topics for every event over the last 6 months.  I'll happily take feedback on what people are most interested in hearing about.

The Coalescing Ring Buffer

For anyone who is interested in what LMAX is up to, and is still following my blog, have a look at this post about the latest tool they've open sourced: the Coalescing Ring Buffer.

QCon Day One

I like QCon London, I really do.  Not only is it on home turf, but, as I've said before, it doesn't just focus on technology, or a set of technologies.

Full disclosure: I've been involved in planning QCon this year.  So this time I know all the thinking, hard work, planning and last-minute changes that go into a conference like this.  And it's a joy to be able to sit in the audience and see the conference that you've helped build.

There are things I took out of today that I want to get down on "paper" now, because I think the next few days will have different themes.

Let's Not Forget About Computer Science
I'm so pleased to see this in a conference!  After documenting and talking about the Disruptor so much last year, I felt it was important for us to go back to our roots a bit, and have some Mechanical Sympathy.  Some of the sessions today brought us back to the school room and had us thinking about the tools we're using.

In my new role I'm doing something I've never had to do before, and that's writing a library that will be used by other developers.  Barbara Liskov's keynote had me thinking hard about "readability over writability" and "design for the case that is used most often".  I also came out slightly depressed that some things hadn't changed in 40 years.  Martin Thompson's talk made me wish I had written my driver performance tests up front, and poked me to continue thinking about our users' performance needs.  I'd like to say Damian Conway's keynote made me want to code in Latin, but in fact it motivated me to get back to learning Spanish.  But it also (unintentionally) backed up a point I've heard before, which is that programming is at least twice as hard to learn if you don't speak English.  Probably worse if you don't speak a European language.

Overall, the message I took from the combination of these sessions is that it's almost more important than ever to get back to basics.  Stop obsessing about frameworks and tools and different flavoured JVM languages.  Start remembering all these things are tools, that they run on machines that work with ones and zeros.  Whatever abstractions we're using, we will write better, cleaner, more performant, fit-for-purpose code if we understand our tools, and their strengths and weaknesses.  One of these tools is even our own brain - understand how it shapes the information and presents us with solutions.

Art Is Awesome (and Useful)
I went a bit off topic to go to two sessions on visual information.  My excuse is that this blog needs more pictures, and I want more visual ways of representing what's going on with MongoDB/the driver.

Heather Willem's session encouraged us to doodle throughout.  In fact, forced us to.  Right up front she adresses the fact that doodling is seen as a lack of attention, as a waste of time.  And I realised, sitting there in the audience with my iPad and stylus, that I did feel guilty drawing away while someone talked at me.  But it was a brilliant exercise in unblocking some of those creative juices, and letting us see the power in visual information.  Perfect is not important, pictures are powerful.

Fernando Orellana is an artist fascinated by building machines to create art.  It was awesome to see an artist who also builds robots and codes stuff to do what he wants.  He said "code is like paint, I use it to create".  See? I knew coding was a creative activity.  He's done some stuff that I probably would have been tempted to say "is that art?", like aiming to create 40 000 play doh cars.  Or making creepy rodents that live in "dumb little suburban homes" out of singing hamster toys.  Go to his website and watch the videos, nothing else can describe it.  I was blown away, it was inspiring.  And useful too for sparking creativity in all of us.

Here's my "subconscious drawing" from the start of the session.  We need more child-like art in our lives.

These two sessions really rekindled my desire to do arty, drawing-y creative stuff.  And helped me see how fear blocks many of us from using this medium.

And...
Sitting in the audience as an attendee, different things jump out at you.  I was shocked, as someone who's been involved in suggesting and selecting speakers, at how few women there were again on this first day.  My thought was, how could this happen with me on the committee??  So I have more sympathy for conference organisers than I did, when facing this tough problem, but more conviction than ever that we need to do something different to showcase different types of role models.

On the other hand, it might be my imagination or wishful thinking, but the overall diversity of the attendees seemed much greater than recent years, which is a Good Thing.

I'm sorry for bringing up this subject again, but you might consider it blind of me not to notice something in a conference I've actually worked on.

But overall
I've been really impressed with the first day of the conference.  Not only was it, of course, a great opportunity to meet new people, catch up with old friends, and generally network, the sessions that I went to were excellent, and have me excited to be working in this industry, at this time.  And to be in a position to hear about it all.  And maybe, just maybe, contribute something of my own.

GDL Presents Women Techmakers with Trisha Gee

I was flattered a couple of weeks ago to be interviewed by Google as part of their women techmaker's series, as it moves over to Europe. In this video I talk about going to Mars, education, planning your career, being a developer, and the impact of technology on our lives. So, not much...