Alex Hudson

Thoughts on Technology, Product, & Strategy

Category: performance

Lean versus Agile

People sometimes ask me about the structure of our internal development team, and to what extent we’re truly “agile”. My response is that we’re actually more “lean”. I happily give examples of some of the key working practices we have. I generally don’t explain the difference between “lean” and “agile”, though.

Sometimes, people use these terms interchangeably. I think this is wrong, but understandable. As a JIRA user, I’m used to it offering a Kanban board to run a scrum sprint. This can be a great choice, but it muddies the waters. Let me take this opportunity to explain my thinking then!

Read More

Academia is apparently unmanageable

There’s a great blog post doing the rounds today, titled “Every attempt to manage academia makes it worse“. Going through a number of examples of metric-based assessment, the conclusion is that standard management practice applied to academic work results in obviously worse outcomes.

At the heart of the argument is an interesting contradiction – that it is possible to assess academic work and show that under a specific regime the results are less good, while simultaneously it is impossible to assess the results of academic work in such a way as to improve it. However, it’s possible to accept a slightly weaker form of the argument – that the practice of measuring while science is being done negatively affects the work in a way that appraising the results post-facto doesn’t. I’m not in a position to really know whether or not this is genuinely the case for academic work, but I’m seeing people apply the same argument to software development, and I truly believe it doesn’t apply.

Read More

Brand & Culture: it’s all about Action

There are a lot of people with strong thoughts about brand and culture, and how the two relate to each other. From conversations I’ve had with others, I thought it high time to put my perspective down in writing.

I have a lot of time for this HBR article, “Brand is Culture, Culture is Brand“. It is absolutely correct to say that you cannot build a brand if your business culture does not / will not support and live that brand, and this is a fault seen so commonly. Business rebrand frequently; and it’s very common to see immediate push-back because the way the business operates doesn’t fly with the new brand at all.

However, I think things have to go deeper than this.

Read More

A short review: The Agile Team Onion

This is a quick and pithy review of Emily Webber’s free e-book, “The Agile Team Onion“. At about 20 pages of content, it’s a concise enough work itself – I personally appreciate the laser-like focus on a single subject; in this case, it’s thinking about the various factors that affect agile team make-up, sizing and interfacing with other people and teams.

Read More

Deadlines, estimates, predictions

Project management is always guaranteed to bring out some strong opinions, and a recent Twitter discussion was no different – but, while the core discussion on Twitter was great, it really deserves a much longer-form treatment. Paul Johnston wrote up his thoughts about getting people to talk about predictions instead of deadlines – and much of it is hard to argue with, but I have a bit of a different perspective.

Read More

Some notes on Serverless design: “macro-function oriented architecture”

Over the past couple of days I’ve been engaged in a Twitter discussion about serverless. The trigger for this was Paul Johnston‘s rather excellent series of posts on his experiences with serverless, wrapped up in this decent overview.

First, what is serverless? You can go over and read Paul’s explanation; my take is that there isn’t really a great definition for this yet. Amazon’s Lambda is the canonical implementation, and as the name kind of gives away, it’s very much a function-oriented environment: there are no EC2 instances to manage or anything like that, you upload some code and that code is executed on reception of an event – then you just pay for the compute time used.

This is the “compute as a utility” concept taken more or less to its ultimate extreme: the problem that Amazon (and the others of that ilk) have in terms of provisioning sufficient compute is relatively well-known, and the price of EC2 is artificially quite high compared to where they would likely want to go: there just is not enough supply. The “end of Moore’s law” is partly to blame; we’re still building software like compute power is doubling every 18 months, and it just isn’t.

Fundamentally, efficiency is increasingly the name of the game, and in particular how to get hardware running more at capacity. There are plenty of EC2 instances around doing very little, there are plenty doing way too much (noisy neighbour syndrome), and what Amazon have figured out is that they’re in a pretty decent place to be able to schedule this workload, so long as they can break it down into the right unit.

This is where serverless comes in. I think that’s a poor name for it, because the lack of server management is a principle benefit, but it’s a side-effect. I would probably prefer macro-function oriented architecture, as a similar but distinct practice to micro-service oriented architecture. Microservices have given rise to discovery and scheduling systems, like Zookeeper and Kubernetes, and this form of thinking is probably primarily responsible for the popularity of Docker. Breaking monolithic designs into modular services, ensuring that they are loosely coupled with well-documented network-oriented APIs, is an entirely sensible practice and not in some small part responsible for the overall success Amazon have had following the famous Bezos edict.

Macrofunction and microservice architectures share many similarities; there is a hard limit on the appropriate scale of each function or service, and the limitation of both resources and capability for each feels like a restriction, but is actually a benefit: with the restrictions in place, more assumptions about the behaviour and requirement of such software can be made, and with more assumptions follow more powerful deployment practices – such as Docker. Indeed, Amazon Lambda can scale your macrofunction significantly – frankly, if you design the thing right, you don’t have to worry about scaling ever again.

However, one weakness Paul has rightly spotted is that this is early days: good practice is really yet to be defined, bad practice is inevitable and difficult to avoid, and people attempting to get the benefits now are also having to figure out the pain points.

It’s worth saying that this architecture will not be for everyone – in fact, if you don’t have some kind of request/response to hook into, frankly it won’t work at all – you’re going to find it very difficult to develop a VPN or other long-lived network service in this environment.

Many of the patterns that should be applied in this scenario are familiar to the twelve-factor aficionado. Functions should be written to be stateless, with persistent data discovered and recorded in other services; configuration is passed in externally; et cetera. Interestingly, no native code is supported – I suggest this is no surprise, given Amazon’s investment in Annapurna and their ARM server line. So, interpreted languages only.

A lot of this reminds me a lot of the under-rated and largely unknown PHP framework, Photon. While this is not immediately obvious – Photon’s raison d’etre is more about being able to run long-lived server processes, which is diametrically different to Lambda – the fundamental requirement to treat code as event-driven and the resulting architecture is very similar. In fact, it was very surprising to me that it doesn’t seem to be possible to subscribe a Lambda handler to an SQS topic – it’s possible to hack this via SNS or polling, but there is no apparent direct mechanism.

It’s difficult to disagree that this is the wave of the future: needing to manage fewer resources makes a lot of sense, being able to forget about security updates and the like is also a major win. It also seems unlikely to me that a Lambda-oriented architecture, if developed in the right way, could ever be much more expensive than a traditional one – and ought to be a lot less in practice.

On Hiring A-Players

“Steve Jobs has a saying that A players hire A players; B players hire C players; and C players hire D players. It doesn’t take long to get to Z players. This trickle-down effect causes bozo explosions in companies.” ― Guy Kawasaki

A few times recently I’ve bumped into what I call the “A-Player Theory”. This is a close relative of the “10x Engineer Theory”, and in its usual formulation states that only the best want to hire their equals or betters: everyone else, for whatever reason, hires down. If only you were brave enough to hire people better than you, you’d be creating great teams in no time!

Like a lot of ideas in the tech arena, it feels like this idea has come from the world of elite sports, and this is no bad thing. When I want to explore ideas about peak performance, it’s natural to look to different types of people and try to apply their ideas to my own. However, I don’t think that that this theory is actually that helpful, and it’s simple to explain why.

First, I think there is a simple problem with the formulation in that hiring is a two-way street: not only does the hirer need to want to find the best people, but the best people will need to be attracted to the hirer. I think this second effect is actually the more important: the quality of the talent available to you, as a hirer, will be directly proportional to the quality of your culture. This is a difficult statement to support in this post, and is something I will explore further in the future, but I do believe that this is the case.

But second, and more importantly, I just don’t think the theory holds in elite sports. Now granted, things are rather different in American elite team sports: for example, team-building is crucial in American Football, but processes like “the draft” are  deliberately designed to ensure that C-teams hire A-players (at least in the beginning), so that some degree of competitive parity is ensured.

I’d like to give some examples from the world of soccer, instead. I’ve chosen this sport for a few reasons – partly, I’m very familiar with it, but mainly because it’s a global sport with the requisite elite element, and sufficient money floating about that players move between teams, leagues and countries with extreme ease. If any teams were going to be calculating about hiring A-players, you’d see them in soccer.

To be fair, there are absolutely teams who do this. The most obvious example is the galáctico policy pursued by Real Madrid. However, this turns out to be pretty rare: teams will transfer players for a variety of different reasons, and rate them in different ways, and while there are large-money transfers for world-class players, these make up a small amount of activity at even the largest clubs as a rule.

The absolute best teams in soccer are not associated with world-class A-players; they’re actually a function of the manager. My favourite example right now is Leicester City: not least as I grew up in Leicester, but (correct at the time of writing this) because they’re flying high at the top of the Premiership with one of my favourite managers, Claudio Ranieri. Their leading scorers, with 36 goals between them, are those huge names “Jamie Vardy” and “Riyad Mahrez” (you will be forgiven if you don’t follow soccer for having never heard of the pair of them).

Vardy at least was a record signing: Leicester paid £1 million for him from a non-league club (and reportedly, the deal could even be worth more). Let’s be clear about this, though: for a premier league player, that sum is an absolute pittance. Mahrez came from a league club to be sure – Ligue 2 AAS Sarcelles – for an undisclosed fee, which again will be small in the scheme of the league (and in fact, both players joined before the team were promoted to the Premiership). Both players now absolutely have a market value in tens of millions.

Leicester did not sign A-players; in fact, they signed a bunch of low-league and non-league players in order to rise out of the Championship, and then made the very sensible decision not to attempt to parachute in Premiership stars, and instead stick with the team they had built. This work is largely attributable to their previous manager, Nigel Pearson, and does not particularly break the mold in many ways – this is a well-trodden path that has seen many clubs rise through the leagues at the hands of an excellent manager (e.g. Nottingham Forest in the Clough era, before which they were largely forgettable and underachieving, or Leeds United under Revie, who started as an awful side that struggled to attract youth players let alone professionals).

If anything, in fact, the record of teams within this league that have gone out to buy the best has been pretty awful – Chelsea have done well but at an absolutely enormous cost, and many teams inflated by A-players have quickly fallen once the money ran out (Newcastle, West Ham, as examples).

The manager with the most enviable record for team-building, of course, must be Alex Ferguson. While his best Manchester United teams were full of world-class players, many of whom had been bought in at some cost, his career was partly defined by the number of A-players he allowed to leave the club. Paul Ince was a huge loss to the club, as was Hughes and Kanchelskis, and (not for the first time in his career) Ferguson came under significant pressure to resign. Instead, he opted (having failed to sign some names) to bring in members of the youth team – names who are now instantly recognisable, like the Nevilles, Beckham, Scholes. Commentator Alan Hansen characterized their opening loss with the immortal words, “You can’t win anything with kids”. He also let go Cantona, and Ronaldo (who left at the height of his powers) brought an almost £70 million profit to the club upon sale. Sure, Ferguson also spent money (and brought in players who didn’t perform – Taibi, Djemba-Djemba, Tosic, Zaha, Bebe, to name but a few – some of them real A-players, like Andersen), but Manchester United were one of the most sustainably successful clubs for a long period of time under his leadership.

All of this brings me back to my central point. It’s not the player that is important; it’s the team – and the decision about who to bring into a team, when, and how, is the responsibility of the manager. It’s very easy to say “I’m going to go and hire only A-players!”, but actually, it’s probably one of the worst things you can do for a team. In football terms, you need balance in a side – people with different strengths, abilities and points of view. Football is notorious for great players who leave one club for another, and become totally anonymous shadows of their former selves: this is not because they’re B- or C-players in disguise, but because if a team doesn’t have a A-player shaped hole, that player will not be able to perform like an A-player.

It is incumbent on the best team leaders to develop the team first and foremost; the aim is that based on the performance of the team others will look back and say, “That is a team of A-players”. It’s easy to state how to do this, but remarkably difficult in practice:

  1. set the right culture for the team, right from the off. This is a whole topic in and of itself, and the culture creates the environment for performance but importantly does not trigger performance itself
  2. where an existing team is in place, plan out how to adapt and grow the people within the team, guided by the culture
  3. for each new member being added to the team, think hard about the gap they’re filling and whether or not they are the right “shape” to fill the gap.

Think about the levers in the team that are used to create and maintain performance. Most managers will automatically turn to metrics and reports; these are amongst the least powerful tools. The crucial factors here are ensuring clarity of purpose, adequacy of tools and resources, autonomy to perform and passion for the work. Passion burns most fiercely when fuelled by success, and as a team leader that is your end goal. Hiring is important, it’s best to start in the best place possible, but I don’t believe it’s anything more than a good start.

“The quality of a person’s life is most often a direct reflection of the expectations of their peer group.” ― Tony Robbins