On Monday, Stephen Chin from Oracle visited me at the 10gen offices as part of his NightHacking tour. In the video we talk about my sessions at JavaOne and the Agile presentation I'm giving at Devoxx, and I do some very basic hacking using the MongoDB Java driver, attempting to showcase gradle at the same time. It was a fun experience, even if it's scary being live-streamed and recorded!
Wednesday, October 31, 2012
On Monday, Stephen Chin from Oracle visited me at the 10gen offices as part of his NightHacking tour. In the video we talk about my sessions at JavaOne and the Agile presentation I'm giving at Devoxx, and I do some very basic hacking using the MongoDB Java driver, attempting to showcase gradle at the same time. It was a fun experience, even if it's scary being live-streamed and recorded!
Tuesday, October 23, 2012
So, I've finished my first full week in the new job and I've learnt lots of new stuff. Which is great, because that's usually why you change jobs.
In a traditional relational databases you have tables, and relationships between those tables are achieved with foreign keys. I'm starting to think of these as something kind of grid-shaped with links between them:
(Yes, I'm experimenting again. This time with my shiny new iPad, a stylus and Penultimate. It's good for ad-hoc drawings, but lacks the precision of the graphics tablet and flexibility of GIMP).
At the very high level, it seems like there are four (ish) types of NoSQL databases:
Column family databases feel to me, as a newbie to the field, similar to key/value, which I'll come on to. I've mostly heard Cassandra used as an example of this type of NoSQL database. I guess the way I think of this, and of course I could be wrong/over-simplifying, is a unique key linked to a set of key/values:
Which I'm translating into groups of key/value pairs, with a the ID as a sort of header:
You need the key in order to look up all the details about me. The way I hear it, it's great for writing data, but it's less flexible for ad-hoc queries.
Key/Value
These types of NoSQL database (e.g. Riak) are pretty much as schema-less as you get - just dump key-value pairs into them. To be honest, the best description I found was on dba.stackexchange.com, so I'm not going to re-write that with my (at this point) limited understanding.
From what I've heard so far, both Key/Value and Column Family databases embrace eventual consistency. I don't know how much of that is a function of their data model and how much is decided by the individual products. For some people eventual consistency is deal-breaker, but in many cases it seems to me that it's just a matter of getting your head around this and designing your application appropriately.
Graph
I came across graph databases when I stumbled across Neo4j, chatting to some of the very smart guys there. A graph database lets you model you data as a series of nodes and relationships. And if I think about it, this is not a massive step from either relational models or object models. It doesn't just apply well to the social networking domain (where it's very easy to think in terms of users and their relationships), in actual fact lots of things we design could be modelled this way. Not having used it, I'm not sure just how much of a mental leap you need to take to start thinking that way, but it seems like it might be a good fit for many problems.
I'd be interested in what the architectural trade-offs in using this model are.
Document
Now MongoDB falls into category four, the document database. And as a NoSQL n00b, this is now the product and area I know most about, and am clearly going to be more excited about since 10gen are indoctrinating me in the MongoDB way.
Documents are a familiar structure for developers, especially if they've been working with JSON. So, a document might be:
To me, this looks like it maps onto to my domain-shaped Object Model more easily than a relational database, which always needs some sort of O-R mapping (whether you do this with hibernate or use Spring to do it yourself, you're still mapping tables into objects and vice versa). What I like about the document format is the nested sub-documents for data that belongs together. In relational databases you often end up denormalising for performance anyway, so why not just accept that up front and have it as part of the thing you're storing?
This does have a cost, of course - nothing is without trade-offs. Every time you request this document, you get the whole lot. You can't have the person without the address. So, you do need to understand the relationships (still) and whether you're usually going to want to get all that data at the same time or whether you might want to make two separate calls.
Which brings me on to another thing which is familiar from relational days - foreign keys. A field in your document can be the ID of another document, so you can follow the links through and retrieve other documents associated with the starting one. Again, there are trade-offs here - each link you follow is a different request to the database. These database requests can be very quick, but if you wanted this data every time, you'd probably want it embeded in your first document to save the additional call. I guess it's a latency vs throughput question really - a single query which returns a chunky document, or multiple queries that return smaller ones.
So schema design is still important in document databases even if you don't have a relational schema. No new technology is an excuse to stop thinking about the problem you're trying to solve and understanding the tradeoffs in design.
One of the advantages, it seems, of something like MongoDB over some of the key/value databases is the ability to write ad-hoc queries and to tune for those queries. The data is structured (it's in a document) and it doesn't have to be in the same structure every time - not every document relating to a person needs all the fields that another person might have. But you can still query for people who have blue cars or people who live in London, or people who's surnames begin with G. If you find yourself doing the same query a number of times, you can add indexes to MongoDB the same way you would a relational database.
Semms like I'm getting into more of the nitty-gritty MongoDB details, so I'll stop there and leave that for another time.
In Summary
Classing a whole swathe of products as "NoSQL" is misleading and confusing. The only thing they all share in common is that they are not traditional relational databases. Other than that, some of them are as different from each other as they are from relational databases. I haven't even mentioned caching technologies - these products have functionality which overlaps with NoSQL databases as well. But even then, the purposes are somewhat different, and not even mutually exclusive.
As with anything, it's really important to understand the strengths and weaknesses of a technology, and the demands of your domain. These different ways of organising data, and different products, are going to perform really well in certain circumstances, and pretty poorly when used in others. Getting an understanding of what those strengths and weaknesses are is going to be important in making the correct product/architecture/design decisions.
None of this information is new, there's a lot of material on the web about the different types of NoSQL databases. I'm writing it more for my own benefit than anything else, my memory is notoriously shocking. For more in-depth (and probably more accurate reading) there's:
I'm learning a lot about these new-fangled NoSQL database thingies. The LMAX architecture was based on keeping everything in memory and reducing the waits for IO - messages were journalled to disk, and reads and writes to the MySQL database were off the critical path. Therefore doing anything radical to the storage side of the architecture was just not high on the list of priorities.
Everything I knew about NoSQL I learnt from the various conferences I've been going to in the last year, and even then that's limited - without a business reason to pursue knowledge I know it'll just leak out of my brain, so I avoid sessions with no immediate applicability to me.
Let's summarise what I knew about NoSQL databases before last week:
I was suspicious of the hype surrounding NoSQL, partly because it's associated with the meaningless marketing term "Big Data" and partly because I'm a cynic that sneers at things that get too popular. Here's what I think when I hear the following terms:
I don't know if it's healthy to be this cynical, but I'm too old to jump on every bandwagon that comes along.
Anyway. Back to the people who now pay my bills.
It's unfortunate that the lack of SQL is the thing that captured the imagination, rather than the lack of tables and a relational structure. SQL was never (in my mind) a particularly evil thing, it's a pretty good language for saying "I want this stuff from this place that fits these criteria", and that's something we're going to have to do at some point whatever the technology.
- They don't use SQL. Who knew?
- There are different flavours. There's a graphy one and key-value things and... others...
- They're "scalable" (yes, yes, it's web scale).
- Some/many/all(?) embrace the idea of eventual consistency
I was suspicious of the hype surrounding NoSQL, partly because it's associated with the meaningless marketing term "Big Data" and partly because I'm a cynic that sneers at things that get too popular. Here's what I think when I hear the following terms:
- Cloud - Fire your systems people and ditch your comms room!
- Big Data - Parse Twitter in order to learn how to read your customer's minds!
- NoSQL - Stop paying Oracle!
- Functional - We couldn't get good enough at mainstream programming languages so we switched to something more difficult!
I don't know if it's healthy to be this cynical, but I'm too old to jump on every bandwagon that comes along.
Anyway. Back to the people who now pay my bills.
It's rather more important that it's the structure of the data that's different in NoSQL databases.
In a traditional relational databases you have tables, and relationships between those tables are achieved with foreign keys. I'm starting to think of these as something kind of grid-shaped with links between them:
 |
| Series of database tables and their relationships. Honest. |
- Column Family
- Key/Value
- Graph
- Document
Column family databases feel to me, as a newbie to the field, similar to key/value, which I'll come on to. I've mostly heard Cassandra used as an example of this type of NoSQL database. I guess the way I think of this, and of course I could be wrong/over-simplifying, is a unique key linked to a set of key/values:
Which I'm translating into groups of key/value pairs, with a the ID as a sort of header:
 |
| Key/value pairs grouped by ID |
Key/Value
These types of NoSQL database (e.g. Riak) are pretty much as schema-less as you get - just dump key-value pairs into them. To be honest, the best description I found was on dba.stackexchange.com, so I'm not going to re-write that with my (at this point) limited understanding.
 |
| Never ending lists of key/values |
Graph
I came across graph databases when I stumbled across Neo4j, chatting to some of the very smart guys there. A graph database lets you model you data as a series of nodes and relationships. And if I think about it, this is not a massive step from either relational models or object models. It doesn't just apply well to the social networking domain (where it's very easy to think in terms of users and their relationships), in actual fact lots of things we design could be modelled this way. Not having used it, I'm not sure just how much of a mental leap you need to take to start thinking that way, but it seems like it might be a good fit for many problems.
 |
| Graph of nodes with annotated relationships |
Document
Now MongoDB falls into category four, the document database. And as a NoSQL n00b, this is now the product and area I know most about, and am clearly going to be more excited about since 10gen are indoctrinating me in the MongoDB way.
Documents are a familiar structure for developers, especially if they've been working with JSON. So, a document might be:
To me, this looks like it maps onto to my domain-shaped Object Model more easily than a relational database, which always needs some sort of O-R mapping (whether you do this with hibernate or use Spring to do it yourself, you're still mapping tables into objects and vice versa). What I like about the document format is the nested sub-documents for data that belongs together. In relational databases you often end up denormalising for performance anyway, so why not just accept that up front and have it as part of the thing you're storing?
 |
| A document with sub-documents. Think XML/JSON. |
Which brings me on to another thing which is familiar from relational days - foreign keys. A field in your document can be the ID of another document, so you can follow the links through and retrieve other documents associated with the starting one. Again, there are trade-offs here - each link you follow is a different request to the database. These database requests can be very quick, but if you wanted this data every time, you'd probably want it embeded in your first document to save the additional call. I guess it's a latency vs throughput question really - a single query which returns a chunky document, or multiple queries that return smaller ones.
 |
| Documents can link to other documents. |
One of the advantages, it seems, of something like MongoDB over some of the key/value databases is the ability to write ad-hoc queries and to tune for those queries. The data is structured (it's in a document) and it doesn't have to be in the same structure every time - not every document relating to a person needs all the fields that another person might have. But you can still query for people who have blue cars or people who live in London, or people who's surnames begin with G. If you find yourself doing the same query a number of times, you can add indexes to MongoDB the same way you would a relational database.
Semms like I'm getting into more of the nitty-gritty MongoDB details, so I'll stop there and leave that for another time.
In Summary
Classing a whole swathe of products as "NoSQL" is misleading and confusing. The only thing they all share in common is that they are not traditional relational databases. Other than that, some of them are as different from each other as they are from relational databases. I haven't even mentioned caching technologies - these products have functionality which overlaps with NoSQL databases as well. But even then, the purposes are somewhat different, and not even mutually exclusive.
As with anything, it's really important to understand the strengths and weaknesses of a technology, and the demands of your domain. These different ways of organising data, and different products, are going to perform really well in certain circumstances, and pretty poorly when used in others. Getting an understanding of what those strengths and weaknesses are is going to be important in making the correct product/architecture/design decisions.
None of this information is new, there's a lot of material on the web about the different types of NoSQL databases. I'm writing it more for my own benefit than anything else, my memory is notoriously shocking. For more in-depth (and probably more accurate reading) there's:
- Martin Fowler's NoSQL Distilled
- ...and his introduction to the subject
- Tim Berglund (@tlberglund) did a great overview of three types at JAX London last week. There's a video of the same content (different conference) here.
- http://nosql-database.org/ appears to list all the products that fall under the massive umbrella, but isn't the most usable of sites.
- And yes, I used Wikipedia. Which is probably where I went wrong...
Monday, October 22, 2012
Time for another ad-hoc update on what I'm up to:
I'm also going to be in New York for training from the 5th - 9th November, but I can't swear to my availability during that time - apparently training is going to take up the days, and the evenings I'm going to try and get a jump start on some actual code.
- Thur Oct 25th: LJC JavaOne Circleback Event
- Thur Nov 1st: The Problem With Women - A Technical Approach
- Wed 14th November - Devoxx: Why Open Source Your Secrets, Agile++: When Agile Goes Well
- Sat 24th November: LJC Open Conference
- Thur 29th November: My first MongoDB Webinar.
I'm also going to be in New York for training from the 5th - 9th November, but I can't swear to my availability during that time - apparently training is going to take up the days, and the evenings I'm going to try and get a jump start on some actual code.
Wednesday, October 17, 2012
Seemed like a quiet conference this year. Not really sure why, maybe it was the layout of the massive (and extremely dark) main room; maybe it was the awkward L-shape of the communal space; or maybe this year people were more interested in listening to the (really very good) sessions rather than participating or meeting other people. Whatever the reason, it felt quiet and almost low-key.
Performance seemed pretty high on the agenda, as you'd expect from a London conference, with a number of things on offer:
Performance seemed pretty high on the agenda, as you'd expect from a London conference, with a number of things on offer:
- A great keynote from Kirk Pepperdine and Martijn Verburg, covering a massive range of things to care about when thinking about performance on the first night
- A high-level talk about Java Performance from yours truly (which I may run again for the LJC if there's interest, but it's more likely to be a one-off)
- A deep dive into writing lock-free coding by Mike Barker
- And a talk from Kirk exploring your GC logs.
It was great to see a number of LJC regulars presenting, especially as my own schedule has been so crazy I haven't seen many of them for a long time. So I missed sessions from Bruce, John, Sandro, Russell, James & Richard, but I heard good things about the sessions and was really pleased to chat to all of them.
The highlight of the conference for me though was Brian Goetz's keynote and subsequent session on lambdas. I've been looking into lambdas because I think it's a really interesting addition to the language and I've heard a lot of noise about them. What I thought was most interesting about Brian's talks though was less the information on what they were and how to use them, and more the challenges that face language designers when they have a language which is used by 10 million developers and has been going for nearly 20 years. Ouch. It's amazing they get anything done, let alone something like lambdas which the language was never designed to support.
In keeping with the new job, I went to a few sessions on the Big Data Con - frankly an unfortunate name I feel. Brendan's Mongo & JVM talk was useful, especially given that I might actually be presenting that at some point. What I'd love to see though is a more interesting story around the Java driver. It seems people believe the Java driver needs a little love.
The other interesting NoSql talk was from Tim Berglund's NoSql Smackdown, which was a really great way of highlighting that the NoSql databases are not all solving the same types of problems. The room was packed and the questions were intelligent, so it seems there's still a lot of interest in this kind of introduction to the technology.
Lessons learned:
- Commuting through Victoria Station sucks. I knew this last year but it's just got worse.
- The iPad + stylus combo is not as precise as the graphics tablet, so I'm probably going back to that for illustrations. But I'd still love to do free-drawing with the iPad on the projector at some point.
- Not everyone can follow the deep-dive tech talks, but they still prefer them to introductory talks, maybe because they feel like they're learning something (well, that's my opinion).
I took practically no photos because I kept forgetting I had my camera. I think it's the weird subterranean effect of the hotel basement. Either that or I've turned into a conference zombie - not an unlikely suggestion. And I've still got Devoxx round the corner...
Thursday, October 11, 2012
The time has come, and I'm moving on from LMAX. I've had an incredible (nearly) four years working for one of the most radical finance firms in the world, during which time I feel I've learnt more than the rest of my work experience put together, and had the pleasure to work with some of the smartest and most interesting people I've ever met.
I've been invited to join 10gen and their MongoDB driver team, a challenge I am really looking forward to. After years in finance and in the IT departments of other organisations, I'm finally working for a product firm, and an open source one. I expect it will be very different from anything else I've been involved in.
I hope this means I will be blogging even more, and that I'll have opportunities to abuse my graphics tablet producing more ridiculous scrawlings. I also hope this will give me an opportunity to meet more people as I travel around.
So, as if this were a goodbye e-mail to the company or an out-of-office reply, I should finish with: any further enquiries about the Disruptor should be addressed to the Google Groups list - there are people on there waaay smarter than me anyway.
I've been invited to join 10gen and their MongoDB driver team, a challenge I am really looking forward to. After years in finance and in the IT departments of other organisations, I'm finally working for a product firm, and an open source one. I expect it will be very different from anything else I've been involved in.
I hope this means I will be blogging even more, and that I'll have opportunities to abuse my graphics tablet producing more ridiculous scrawlings. I also hope this will give me an opportunity to meet more people as I travel around.
So, as if this were a goodbye e-mail to the company or an out-of-office reply, I should finish with: any further enquiries about the Disruptor should be addressed to the Google Groups list - there are people on there waaay smarter than me anyway.
 |
| Lots of shiny new goodies! |
Wednesday, October 10, 2012
I've produced a very cut down version of the presentation I've been giving at a lot of conferences, giving a high level overview to the Disruptor. This serves as a quick intro to the concepts behind it.
My slides are usually pretty useless without me (or someone else) talking over them, so for more context don't forget there's always my original blog posts (the Magic Ring Buffer, Reading from it, Writing to it, Wiring it up), which are now pretty dated, and the Java Magazine article I wrote at the start of the year.
My slides are usually pretty useless without me (or someone else) talking over them, so for more context don't forget there's always my original blog posts (the Magic Ring Buffer, Reading from it, Writing to it, Wiring it up), which are now pretty dated, and the Java Magazine article I wrote at the start of the year.
Friday, October 5, 2012
 |
| Taylor Street Cafe |
Maybe it's because since I first spoke at JavaOne last year I've met a lot of people in this ecosystem - Oracle Java people, other conference organisers, Java User Group leaders, speakers and attendees from all sorts of places - and seeing many of them in one place is fun and makes you feel like you're "home". I'd be sat in the Taylor St Cafe area and people would sit down and hang out with me. I'd be hiding somewhere trying to write a presentation and people would say Hi. I'd be separated from the other LJC guys while I grabbed a glass of water and someone would stop to chat. It was awesome. I even managed to get a tiny bit of work done between socialising.
Personally, I have learnt very valuable lessons:
- Doing two conferences back to back is exhausting. If I do it ever again, I really need to optimise my travel between them (i.e. not fly from St Louis to London and then to San Francisco 48 hours later).
- Especially if I have two conferences back to back, but almost definitely as a rule, it's best to have the materials for presentations complete before the conference. I probably would have enjoyed myself even more, and definitely had a chance to see a lot more sessions, if I hadn't had one presentation to finish.
- Presenting four sessions in one conference is extremely challenging. Technically I only had two full-length presentations (the Disruptor presentation I've done at OSCON/GOTO/QCon etc and my Technical Approach to Women, which was more like a very structured BOF), a short 20-minute talk and a panel. But I think one full-length presentation and a couple of other bits is probably a much more sane amount. Especially given 1) and 2) above.
There will be a bunch of other blog posts mentioning the inconvenience of having it in three hotels, the poor suitability of some of the rooms, the difficulty of sharing San Francisco with Oracle Open World, the mediocre food. But you don't spend the money to fly halfway across the world to San Francisco for the crappy box lunches. You fly here to meet - face-to-face - some of the people who created Java, those who are steering it now, and the community that continues to make it relevant and fun.
Highlights for me:
 |
| James Gosling talks about Robots! |
- Technical keynote
- Dr Robert Ballard's keynote
- Community keynote
- James Gosling on stage!
- Having well-known Java people say Hi to me out of the blue
- My Women in Technology session
I already mentioned the technical keynote in my summary of Sunday, but I forgot to talk about Dr Robert Ballard's section, which is crazy because it was so inspiring. What I got out of that was how much fun technology can be, and how kids are really ready to be amazed and engaged with science and technology, provided you get them young enough. I also really loved that the photos of all the scientists showed great diversity, something he seems to take very seriously. His comment was "A child needs to see their face 20 years out to know they can play in the game. Don't sell science, sell scientists and engineers". The point I took from that was that by focussing on "science" and "technology", we could be losing people - by showcasing the people, the talent, the passions, we are much more likely to engage children. And I think that stands for adults as well.
 |
| A robot! |
And this nicely fits the message I was trying to get across in my session on Wednesday (which I know I've already blogged about) - it's dead important to address the failure points when it comes to diversity, but I think we forget to push the positives - the reason we don't understand why some groups of people are under-represented in technology is because we all love being here, it's a great industry to work in. I'd love to see our passions more widely showcased than simply at conferences where we're a self-selecting audience.
 |
| The LJC panel about our involvement in the Java Community Process |
Not only was this a great JavaOne for me personally, it was a real triumph for the London Java Community. Last year there were only two of us here, and Martijn was working hard to introduce me to the other JUG leaders and the people on the JCP committee that we were going to be working with over the next 12 months. This year, there were five of us here, spreading the word about our work as a member of the Executive Committee of the Java Community Process (the Java standards body), primarily our Adopt a JSR program. We also wanted to talk to a load of people about the Adopt OpenJDK program as well, we've found lots of developers are really keen to develop the language itself, Java at the fundamental level, and we really want to help bridge the gap between the guys who work on the OpenJDK project and the developers in the community. But all our hard work over the last year, kicking off these programs and working with other user groups and Oracle to really harness the power of the community, has been recognised. We've won not one, but two awards - woohoo! Fame, glory, girls/boys, money etc await us! Well, not quite... but it's certainly nice to be the "Award Winning London Java Community".
 |
| The LJC in the Community Keynote |
Our two shiny accolades are:
- Duke's Choice Award for Innovation and Contributions to the Community
- JCP Member of the Year with SouJava
So, a really great JavaOne. I'm not sure, if I'm honest, who reads my summaries of conferences - I know lots of people do read them, but I don't know if that's those who went to the conference or those who would have liked to have gone, or those who don't know what it's like at conferences.
But in case this is useful to anyone who is reading this, next year come to JavaOne if:
- You want to know what's coming up for Java the platform or Java the language
- You want to meet, or at least see presenting, the people who are actually making these changes
- You want to meet other Java developers from around the world - whether you want to find someone with the same problems as you, or want to see different issues that face people.
- You want to see the Power Of User Groups!
- You want to understand how decisions are made with regards to the future of Java
- You want to meet like-minded people
 |
| The Exhibitors Hall was like Vegas! But there was a buzz which was infectious |
Oh there's millions more, and even if you only tick one of those reasons, I personally think it's a great way to get a better understanding of the language we use every day as developers.
And finally, some of my sessions will be available, audio and slides only, via the content catalog. Look for the following:
- UGF10467 - Benefits of Open Source
- CON3732 - Concurrent Programming with the Disruptor
- CON5130 - London Java Community: How to Change the World
- CON11338 - The Problem with Women: A Technical Approach
Subscribe to:
Posts (Atom)
