Overheard: Agile truths

After attending a number of conferences and events, and performing numerous interviews, I'm starting to hear the same things again and again.  Since Dan North challenged all my assumptions at QCon, I'm reluctant to outright ridicule them, but I will put forward my personal opinion.

Note: these are things I have heard from multiple sources, so with any luck I am not breaking the sanctity of the confessional interview.

I've never pair programmed, but I've frequently worked with a partner on critical production problems
I find this fascinating.  If there's one thing that needs to be fixed as fast, as correctly, as efficiently as possible, it's a production issue.  And when there is one, "everyone" knows that two heads are better than one, even The Business.

If this is the case, why is it so hard to sell pair programming as the default state of affairs?

Is it because creating new features is seen as just typing, where the bottleneck is access to the physical keyboard?  Is it because fixing defects when the pressure isn't on is suddenly easier for one person on their own without help?

This state of affairs is interesting to me as it implies that when proverbial hits the fan, the instinctive thing to do is to work collaboratively.  Why don't we do it more often?

We use Test Driven Development to get coverage
Seems weird to me to write your tests first to get coverage.  If unit test coverage is your most important metric (and other people have covered why this might not be the case), I'm not sure why you would write your tests first.  Seems to me that you'd get better coverage writing the tests after the code.  That way you can be sure you've covered every eventuality.

To me, the statement implies two assumptions which I would challenge:
a) The primary value of writing your tests first is to meet your coverage requirements
b) Coverage is a meaningful metric

TDD/BDD has a number of benefits (...and now I'm reluctant to list them here in case people repeat them back to me in an interview).  Good coverage will probably be a side effect of being forced to write your tests first, but I'm not convinced that's the best thing that will come out of using TDD.

I only test first when I know what I want to code
I've overheard people saying that they test first when they know what the code is going to look like.  So you dive straight into the code when you don't know what you're doing???

Of course there is a place for this - spikes, prototyping, getting a feel for a new library, so on and so forth.  But I feel that for most code that you write in your day job, you probably have a business requirement and possibly (probably?) a less firm idea of how you're going to code it.  To me, this translates into writing the test first (which documents what you want to deliver, which you already know) and then getting that to pass (which is writing the code, which is the bit you might not know).

If you know exactly what the code is going to look like a) I would question that statement and b) what's the point of the test?


What are the real answers?
At QCon I saw on Twitter a number of complaints because the presentations there gave opinions, guidelines, and, worst of all, a lot of "it depends".  But people seemed to want The Answers.

In my opinion, what developers get paid for is working out the "it depends" parameters and selecting an approach, technical or process-wise, that works for their situation.

So although I have strong opinions on all the above subjects, and although LMAX has specific approaches to both pair programming and automated testing, sadly I'm not going to go into lots of details about those.

Mostly because I'm still interviewing candidates, and I don't want to give away the correct answers....

QCon London 2012

I'm late with my write-up of QCon, and what's worse, it will be partial - "sadly" I was in Lanzarote on a training week with the running club from the Thursday (8th) so I missed most of it.  A sacrifice I had to make for 7 days in the sunshine….

Firstly, me me me

I presented the talk I previewed at Skillsmatter the previous week, something I was calling the User's Guide to the Disruptor, but actually turned out to be how-can-Trish-fill-95-slides-with-pictures-and-finish-in-under-40-minutes.

The audience was different to the Skillsmatter event, not surprisingly.  What was surprising is that I expected people at the conference to be less aware of the Disruptor, and those who came to the Disruptor-only LJC event to have had more exposure to it.  It was a (pleasant) surprise to see how many of the standing-room-only audience had not only heard of the Disruptor but had read stuff about it (I always love it when people have read my blog), played with it and were even using it in anger.

Because of that, I think if anything the talk did not go into enough detail, or enough new stuff, to please everyone.  Tough crowd!  But it was gratifying to hear the audience correct me in some of my answers, and answer other people's questions - it's always nice to know people are listening.

Of course, I will post a link to the presentation when it's available.  For now only the slides are online.

I enjoyed QCon

QCon was noticeably different to the other conferences I've been to in the last six months.  For one, it's not a Java conference - sure, I was hosted on the Java track, but QCon is wider-ranging than just one technology platform.  I'm not sure if it's because of this, or because it was based in the notoriously impatient London, but I felt like there was a message of "Look, let's just get stuff done, OK?".  Ultimately we get paid to deliver stuff for the business, and since my favourite question is always "but what are we trying to achieve?" I like to hear ideas around how to actually deliver.  Don't get me wrong, I've loved the technology conferences - I like to hear about new stuff I had no idea about, and I really like the community vibe from them.  But it's a nice change to be shaken up into thinking exactly why we do all this.

The Data Panorama

Firstly, Rebecca Parsons and Martin Fowler from my old employer ThoughtWorks put Big Data into perspective.  Previously I hadn't really cared about it - we process lots of data at LMAX but we don't really have to dig into it, so Big Data is not top of my "oooh I really worry about that" problems.  There were quite a few interesting points I took out of this:

 - In the past, it was easy (and possibly even correct) to model the whole application based on the data you were collecting or manipulating (and probably storing in a relational database).  These days it's not just the data from your app you need to worry about (and that can get big enough), but also all the news, blogs, twitter, and Facebook stuff generated by you and about you.  In addition, your data might not even be located with your app - the cloud has made the physical question of location redundant.  All of this pushes you towards an architecture which has to separate data from the application, and forces you to ponder your design.  I heard good arguments for Domain Driven Design here, which is nice because we like that at LMAX.

 - Reporting and analytics on Big Data must be more fluid.  There's so much data about you, your users, your application, out there that you don't even know the questions to ask.  Instead you want to be able to spot patterns in data you didn't even know you had,  I thought this was dead interesting - I studied Computer Science and Artificial Intelligence at university, and we were told data mining using AI was going to be fundamental to companies who wanted to be on the bleeding edge.  Only now is it looking like people realise it's becoming that important.

 - Martin referred to Data Scientists and said he was suspicious of scientists.  I'd been reading The Black Swan on the train in that morning and couldn't help but wonder if he's read the same thing - that was talking about how you should treat someone with suspicion if they suggest you can apply science and logic to anything that is… well, actually to anything other than actual science (by which he meant physics I believe).  By saying it's a science you suggest it's predictable and follows rules.  And if it was predictable, it wouldn't be hard.  Big Data is anything other than predictable - the data could be corrupt (it's safest to assume some of it is); it's generated by people (and we all know how fickle they are); and if you're collecting lots of it practically randomly, then cause/effect/correlation are not guaranteed.  So Martin suggests the term Data Journalist.  There was a storm on Twitter which suggested a certain amount of disagreement with this term, but I like it.  But then, I like to pretend I'm a writer and not a programmer.

 - They gave some examples of using data to drive economic growth (e.g. Kenya) - what I thought was interesting about this is that it was a win-win situation - expose the data to grow technology skills, but get a lot of interesting/useful/socially responsible applications in return.

 - Something I liked was about the idea of embracing "bad" data, stuff that's not trustworthy for whatever reason.  You can assume that the good data overwhelms the bad but you can't be sure.  By looking for more fluffy patterns, vague correlations, in your data, you might expose something interesting even if the data's not "correct".  As humans, we can't expect that we won't make a mistake - it's better assume we will and work out how to deal with it.

 - There was a call to not passively accept requirements, but to play an active role.  I like to think this ties in to my post about working with your customer.  But then I would.

 - I got a lot from this keynote, even if it was just a feeling of "I knew it!  I was right!". 

Highly available systems in Erlang

Joe Armstrong was a very interesting person to listen to, clearly someone who's been there and done that. Even without the presentation, the slides are interesting in their own right as they contain a lot of information and guidelines.

The points I took away are:

 - If part of the system fails, it's not up to that part to fix itself.  You need special help to deal with failures.  If you fell over with a heart attack, you wouldn't try and heal yourself, you'd get a medic

 - Isolation between threads/programs will mean that those different things cannot interfere with each other (i.e. no shared state).

 - My favourite quote was "If you make things synchronous you'll bugger things up".  In theory, it's so much easier for us humans (programmers) to think synchronously.  But whenever you design your system to by asynchronous, you find that your system actually becomes simpler and not more complicated.

JVM performance optimizations at Twitter's scale

 - I had a terrible view in the fully packed room, so I was just picking up phrases. I heard Attila mention the Uncanny Valley, a phenomenon I was introduced to by my sister when she was studying her Cybernetics PhD (Google it, I found it fascinating).  

 - There was a lot of really useful information about how the Java GC works.  It seemed to back up the (rough) premise we work on - stuff that's very short-lived is fine, and stuff that lasts "forever" is fine - it's the stuff in between which  cripples your system when it keeps getting shoved around.

Decisions Decisions

Dan North was, as always, an excellent speaker.  He entertained us but got us all thinking.  Five years ago at QCon London (my very first conference ever, and the thing that motivated me to start a professional blog), I saw Dan speaking and I was inspired to think about my working practices.  He was talking about BDD at the time, which was a relatively new concept to me.  I came away from that QCon with clearer ideas of what awesome development practices should look like.  Never did I imagine that five years later, not only would I be working in an environment which follows a lot of those practices (and pioneers many more) but that I would actually be speaking at the same conference.

I've come a long way since then, and of course, Dan isn't talking about the same things either.  I've been to a lot of conferences this year, and I heard nothing really preaching to the "meta-agile" LMAX (still have a post pending about our agile practices...).  When it comes to agile, there seem to be very few people who we can learn off - don't get me wrong, there are lots of things we want to improve on, which is why we're looking for people to learn off.  But most people are still preaching TDD and we want to know "what next?".

Well, Dan took everything we thought we knew, and ripped it to pieces.  Taught us to challenge everything we think we know.

It was irritating actually because he had no answers.  But he did tell us that the answers we think we already have might not be the right ones….

Well worth watching his talk when it comes online, I really can't summarise it here.

Developers Have a Mental Disorder

I nearly missed this ending keynote.  I'm so glad I didn't, Greg Young is awesome at ranting.  He said developers have a disease - we overcomplicate things, when we try to simplify things.  We want to abstract stuff, we love to look for patterns and reuse when actually sometimes we just need to solve the problem.

In my notes I have "People come to conferences for answers, when they should be remembering to use their brains".  I assume that's a quote from him, and not a comment I thought of at the time, but I wholeheartedly agree with it - if a shiny new technology solved your problems, you'd be out of a job.  Your job is to take the problem and figure out how to solve it.  Not to drag and drop an answer into place.

Another talk that I can't do justice to, watch the video when it comes online.

…and finally….

At the end of the day, the Atlassian-sponsored community night was awesome.  I got to chat to (be prepared for gratuitous name dropping here) Martin Fowler; my old friend Simon Brown; my ubiquitous LJC colleagues Martijn and John; the Zero Turnaround guys; a couple of ex-colleagues from Evolution / Detica; a heap of LMAX and ex-LMAX guys; and, of course, bundles and bundles of new interesting people.

Summary (i.e the short version for those who can't be bothered to read this whole post)

Maybe it's wishful thinking, but the messages I took from QCon were:

• If you want to solve the problem your business has, you might want to model your system around their world.  Funny, that sounds suspiciously like Domain Driven Design.
• Synchronous is bad, mmmkay? 
• Hardcore understanding of what the computer is really doing seems to be coming back into fashion.  Hmmm, I wonder who started that…? (tongue firmly in cheek, we can't have been the only ones)
• "You can't tune something you don't understand" - testing and monitoring is kinda important.
• Our business is all about trade offs.  There is no perfect solution, they pay us because it's very very hard to work out something that's Good Enough.
• Ultimately it feels like back to basics: understand the problem; model the domain, and have sympathy for the hardware that's running your solution.

My Corporate Bit

QCon overall turned out to be a bit of an LMAX fest in the end, with Mike & Martin and Andy Stewart (our Chief Lord Business Analyst), all giving presentations there as well as me.  It's nice to be on home turf, and it's very cool to see that we have such a range of things to talk about that so many of us are invited to speak.

PS

I just found out there's a QCon in New York.  My invitation seems to have got lost in the post.  Don't suppose anyone wants me to speak there…?

Java Magazine: Intro to the Disruptor Part One

This month's Java Magazine features an article by yours truly, which is yet another intro to the Disruptor.  It's basically a summary of the stuff I've written in this blog, updated for version 2.7 - so the names of the classes should be up to date and the responsibilities follow the simplified pattern we use now.  If you were looking for an more recent version of my introduction blog posts, this article gives a reasonable overview.

This is intended as part one of a series, as it's a basic and high-level view with no code examples.  In fact, it probably could be used to document the C# version as well as the Java version, although I haven't taken a look at that for a while.  Next, I would like to give some more code examples of how you use it - as always, any suggestions welcome.

New Disruptor Presentation Unveiled to the LJC!

A few weeks ago, I presented my new "User's Guide to the Disruptor" talk to the London Java Community.  Since it was very kindly hosted at Skillsmatter, there is a video of the presentation available, and the slides are below.

Concurrent Programming Using the Disruptor

View more presentations from Trisha Gee

The presentation is a little different to the ones we've done before.  Previously we've gone into how it works and why it's fast.  This time I wanted to step back a little from the internals and show how real developers might actually use it.  The example is somewhat contrived, but the idea is to give some hints on how to break your problem down into something that will work with a Disruptor at the heart of it.

I thought the event went really well.  It was a tiny bit completely intimidating, as there were no lightning talks and I was the only attraction.  Seeing over a hundred people turn up after work, before beer, to hear you talk is a humbling experience.  Fortunately, I think the audience was perfect for the presentation - they had heard of the Disruptor but hadn't seen anything very detailed about it, so my walkthrough of how you might use it seemed to go down well.  I certainly got a lot of very sensible questions (which hopefully I've remembered to repeat for the benefit of the recording), and people had some good ideas about how and when to use it.

I ran the same presentation to a different audience at QCon London a week later, I'll post a link to that if/when it becomes available.

Why the customer isn't always right

Last week I went to get my hair cut (yes, sorry, this is a story about hair).  I had thought long and hard about what I wanted.  I researched, checked styles online, and bought a magazine so I could show my hairdresser exactly what I was after and there would be no confusion.  I was determined I would not be spending that ridiculous amount of money on something I was not going to be happy with.  I was even bold enough to ask for some changes to it at the end, which I have never ever had the courage to do before.

He did an excellent job.  It was almost exactly what I had asked for, with some variations to account for my particular hair type.  It was a very cute hair style that suited me.  But I had a niggling doubt.

A few days later, that niggle was a certainty.  It wasn't what I wanted.

However, it was what I had asked for.

Being English, the thought of going back and telling him I wasn't happy with it was horrifying.  Especially since he had done a really good job of it.  It wasn't his fault that what I'd asked for wasn't what I had actually wanted.  But I knew what I wanted now, and I was prepared to pay for it (again).

This time I didn't show him a picture.  I didn't point to anything specific.  I said I wanted it much shorter after all and outlined the look I was going for.  We had a conversation about it and I left it to him to apply his skills to actually implement it.

This time, I was much happier with the results.  In fact, it was exactly what I wanted.

So what?

Of course this got me thinking about work.  It's not just a lame excuse to write about girl stuff on a supposedly technical blog.  It got me thinking about our customers.

Whoever our customers are, whether they're end users, internal business owners, external clients, or we work in a bank and have fifteen thousand layers of Business Analysts between us and Real People, whoever they are they want something from us.  And in many places, we, the techies, have trained them to ask for more and more specific things, so they can be sure they're going to get what they really want.  Remember that time Bob asked for that extra field on that report because it would make his job easier?  Remember when Sandra wanted the workflow to be altered in a specific way?  It's because they know that if they ask for something specific, the chances are better that it will make its way to the front of the work queue at some point, and when it comes out the other end it is more likely to be what they wanted.

Only it isn't, is it?

There's always another field to add, or another change to make.  And your customer isn't quite satisfied, even though you did exactly what they asked for.  And sometimes they don't tell you that they're not getting what they need from your system, because it's expensive or time consuming to get changes made.

The key to delivering something they really want, instead of something else that's merely adding to the weight of your code, is finding out what they're actually trying to achieve.

So Bob wants that extra field.  That's easy enough.  But when you ask him why he wants that extra field, turns out it's because the numbers in the reports he's getting at the moment aren't the ones he needs, or aren't quite correct.  He's probably created a monstrous Excel spreadsheet into which he manually types half of the numbers from the reports it took you ages to develop, which munges them into something quite different, in order to get the real thing he wants.  He's just missing this one small piece of information to get the final figure correct.  It would save your company a lot of time/money/mis-typing errors if you completely re-wrote the reports Bob uses to give him exactly what he needs.  Or if you gave him a tool to download CSV formatted raw data from the database.

It's not Bob's fault he can't ask for this, he doesn't know what we're capable of providing him.  It's not our fault for not delivering it the first time round, we don't know what he does day-to-day.  But having a conversation where we recognise where the knowledge lies (Bob knows what the output of his day job is presumably, and we know what's available and how we can provide it), and collaborating to come up with the least rubbish solution for everyone, is a step to providing that overused term, "business value".

We're not on different sides here, we all play for the same team - we want our jobs to be as painless as possible, which (should) ultimately provide better efficiency for our company.  After all, we want the company to make money so they can pay us, right?



Caveat: Mileage may vary.  Sometimes, with some customers, you really do need to tell them what they want.

Upcoming speaking events

In theory, I am busy writing material for my upcoming speaking events, rather than writing terribly illuminating posts on my blog (see what I did there?).  In actuality I am being lazy and have pretty much taken January off for a recharge.

In the spirit of doing something which ticks both the event-speaking and blogging boxes, this is a quick update on the conferences I'm confirmed for so far.  Put the following dates in your diary - these are my first international solo speaking events:

7th March - QCon London - Concurrent Programming Using The Disruptor (sadly I can't stay for the whole conference as it clashes with the only holiday I had booked for 2012).
23rd May - GOTO Copenhagen - Concurrent Programming Using The Disruptor & War Stories.
25-26th May - GOTO Amsterdam - Concurrent Programming Using The Disruptor.

The presentation will be more of a user's guide to the Disruptor than anything we've done before.  An hour isn't a lot of time to cover all the functionality everyone might want to see, so I'm still trying to work out the balance between giving an introduction/overview for those who haven't seen it before, and going into some of the cool features that have been added since I first started blogging about it.  If there's anything you would particularly like to see covered, let me know - I'll put the most frequently requested things in there.

Ideally I'd run a workshop session at some point, but that will require quite a lot more preparation, so I'll only do that if there is interest in it (if someone wants to fly me somewhere interesting to do that so much the better!!).

Maybe I'll see you at one of these events?

Christmas decorations teach me a lesson about troubleshooting

And now, after an absence of several weeks, you get to see how long it takes me to write some of these posts.

I was putting up the Christmas decorations one Saturday when my worst fear was realised¹ - one of my three strings of lights was not working.

The first two went up fine.  The third lit up when I plugged it in, and in less than a second went out.  Curses.  This is not what I wanted, this was supposed to be a short exercise in making my tiny little flat look festive.

So I set about the tedious task of starting from the end closest to the plug and replacing every bulb, one by one, with a spare one to see if it magically lit up again.  When it doesn't, you take the spare back out and replace it with the original bulb.  I remember my parents going through this ritual every Christmas, the tediousness of this activity is more memorable than the fleeting joy of shinies.

While I was doing this, my mind was back on the job I'd been doing at work the previous week - battling an Internet Explorer 7 performance problem.  We have automated performance tests which give us an indication of the load time for our application in Chrome and IE, and some time in the previous couple of weeks our IE performance had significantly degraded in the development code.  Due to a number of too-boring-to-explain-here circumstances, the last known good revision was four days and nearly 250 revisions earlier than the first revision that showed the performance problem.

Since we couldn't see anything to indicate it was an environmental problem, the logical next step was to pinpoint the revision which caused the problem, so we could either fix it or get performance gains from somewhere else in the system.

The most obvious way to do this, given there were no obvious suspects, is with a binary search of the revisions.  Our last known good revision was 081, our first poor performing one was 240.  So the thing to do is to check revision 160, see if it falls on the poor or good performance side.

If 160 proves to be a poor performer, check revision 120....

...if 160 is fine, test revision 200...

...and keep splitting the revisions by half until you find the suspect.

So of course that's what I want to do with my stupid Christmas lights.  I do not want to sequentially check each light bulb, that has a worst-case number-of-bulbs-tried = n, where n is the number of bulbs (probably a couple of hundred, although it felt like several thousand).  So, in computer speak, O(n).  The binary search algorithm is O(log n).  At university, this Big Oh had no context for me.  But when you've taken 10 minutes to get a quarter of the way through your Christmas lights, and you diagnosed your IE performance problems... well, actually it took days.  But the point is, a binary search for the missing bulb would definitely have been a Good Thing.

I know you're dying to know if I tracked down the problem in Internet Explorer - I did.  What's the worst case when you're doing a binary search?  It's when the thing you're looking for is veeeery close to either your start point or your end point.

The revision number I was after was number 237.  Sigh.

And my Christmas tree lights?  Well, through the boredom I remembered that modern lights are in sections, so they have a sort of built-in binary search - well, limited segments will go dark if a single bulb is out - which allows you to narrow down your problem area.  Since the whole string was out, I figured something else was probably wrong.

Turned out the plug had come out of the socket.

So:
Lesson 1: Theoretical computer science does have a place when you care about how long something takes.  When it's you feeling the pain, you'll do anything to make it stop.

Lesson 2: When diagnosing a problem you will always biased towards what you think it is, in the face of actual evidence.  I was afraid I would have to search the whole set of lights for a blown bulb, so that's the problem I was looking for when the lights failed.  In actual fact it was a problem with a much simpler solution.



¹ - OK, "worst fear" in this very limited context only - it's not like I lie awake at night in July afraid that one of the bulbs on my Christmas lights has blown.