Full transcript
0:04all right everybody thank you coming
0:07this afternoon
0:08last breakout of the day and then we get
0:10to go listen to stories about Pixar
0:12which I'm looking forward to my name is
0:15Brandon keepers I'm the open source lead
0:18at github and today I want to talk about
0:21how open source principles can apply to
0:24our own engineering teams so there's no
0:28doubt that as an industry we witness the
0:30success of open source software we've
0:33seen the open source process create
0:37extremely high quality systems the scale
0:41to some you know massive levels and this
0:45is applied to systems like you know
0:47operating systems or distributed
0:49databases or web browsers pieces of
0:52software that are not simple and so
0:55today I want to talk about what some of
0:57the the principles behind open source
0:59are the net that may lead to those
1:02designs and think about how we can apply
1:04some of those to our internal teams what
1:08I hope you take away from this session
1:10today are is somewhat somewhat of a
1:13framework to think about how you can
1:15apply these principles to your team not
1:17necessarily you know there be a few
1:19specific tactics you can use but I'm
1:22looking forward to the ways that you can
1:23see how these principles fit into your
1:25own team so
1:32kind of the general principles of open
1:34source are our transparency
1:35participation and collaboration when you
1:38hear when you hear people outside of the
1:40software world talk about applying
1:42open-source to their realm these are the
1:44three things they're talking about so so
1:47whether it's open government or open
1:48data open society any anybody that's
1:51seen the success of open source and
1:53wants to figure out how to mimic it what
1:55they're talking about is we want to
1:56figure out how to bring these attributes
1:57to it so we want to we want to expose
2:00processes and information make them
2:02transparent we want to enable people to
2:04participate and then we hopefully
2:08through that participation we can
2:10actually collaborate and build something
2:11useful together and so today I want to
2:16talk about how I think that these
2:17principles have somewhat of a cascading
2:19effect yeah you can't have collaboration
2:22without people participating you can't
2:23have participation without transparency
2:26now no and specifically I want to talk
2:28about the communication structures that
2:30I think create these things so let's
2:35start with transparency obviously in
2:39open source the source code is open
2:43right it's available for anyone to its
2:46inspect you can take it apart you can
2:47examine it you can even fix it you can
2:50learn about good and bad design patterns
2:53but that's not necessarily what we're
2:56going to talk about today I mean I doubt
2:58that most of us aren't into internal
2:59teams want to release the source code
3:02but the reality is that that's not
3:04actually where the value of open source
3:06software comes some people do do those
3:08things some people look at the source
3:09code and take it apart some people learn
3:11from it but most people don't but we
3:13still benefit from the value of that
3:15source code being available the source
3:19code is an artifact of a production
3:21process and what's interesting about
3:24production processes is they generally
3:26have a larger impact than the thing that
3:27they create like once you once you
3:29figure way of making things the entire
3:32world can take that process and apply it
3:34to the things that they make so we saw
3:36this with you know Henry Ford in the
3:39automobile like it wasn't the fact that
3:41Henry Ford made a better car he figured
3:43out a better way to make
3:44and that thus enabled an entire industry
3:47to take off and so that this artifact
3:51again is not of a specific process it's
3:53not of a you know saw the development
3:56methodology it's not you know the result
3:59of understanding a specific users needs
4:02but I would argue that it's the
4:04communication structure so in the late
4:0760s there was a guy named Melvin Conway
4:10who wrote this paper called how
4:12committees invent and that the central
4:15thesis of that paper was any
4:16organization that designs a system will
4:19inevitably produce a design who
4:20structures a copy of the organization's
4:22communication structure I think this is
4:25really interesting kind of to give an
4:27example of this you know if you took a
4:30traditional team of software developers
4:31say five people and you ask them to
4:34design a system according to Conway's
4:36law you'd basically end up with a system
4:39that had at least four parts and I say
4:41four because you'd have four engineers
4:43doing work and then one manager
4:44theoretically um so the example that
4:47it's often given if you know if you
4:49asked them to write a compiler you'd end
4:51up with a four phase compiler and it
4:54would work because the way that this
4:55team communicates you have you know
4:57manager that kind of ties everything
4:58together but there would be at least
4:59four distinct components in that so I
5:03think that this you know this is a
5:04really interesting observation the paper
5:06that he wrote is very short so I
5:07encourage you to go read it but it's had
5:09a huge impact on some of the thinking of
5:11you know development methodologies and
5:14like the book the mythical man-month by
5:16Fred books Brooks cites this and kind of
5:19uses it to form its thinking so if we
5:24look at the way that open-source
5:26communities communicate we first have to
5:29understand strengths that they have so
5:32the open source communities are
5:34distributed geographically across time
5:37zones and even across cultures this
5:40means that they don't have the benefit
5:41of you know high fidelity communication
5:43right you can't walk over and tap
5:45somebody on the shoulder when you're
5:46stuck on a problem you can't jump into a
5:49meeting when you're trying to work
5:50through it through a problem you can't
5:53even know stand around the water cooler
5:54and chat just to get to know each other
5:58so this means that these projects have
6:00to accept certain constraints in in
6:04their workflow so several years ago Ryan
6:06Sumiko who was actually the original
6:08github engineer the first engineer hired
6:10at github talked about a lot of these so
6:12so a lot of what I'm going to talk about
6:14next comes from him and then also Ben
6:16Balter who's giving a talk right now and
6:18github pages has done a lot of work and
6:20trying to articulate some of these ideas
6:25so open source projects used this
6:28basically to use tools that enforce this
6:31communication structure the medium is
6:33electronic it's asynchronous it's lock
6:38free and the result of those three
6:40things is that the message is archived
6:42so let's dive into to what that means so
6:46the medium is electronic open-source
6:48communities opt obviously for tools that
6:51are electronic the Internet and
6:53Technology in general has played a huge
6:55role and even enabling this like massive
6:58collaboration so you have things like
7:00issue tracker which is just this ongoing
7:02list of either possible improvements
7:04bugs ideas and this this issue tracker
7:09is open to anybody that wants to get
7:11involved in the project you can go in
7:13and look at all the discussions that
7:14have happened in the entire history of
7:16the project it kind of creates this like
7:18corpus of possible improvements for the
7:20software you have version control which
7:23I think most of us take for granted but
7:25most most other professions don't have
7:30the ability or even you know any any
7:32form of tool that will let them go off
7:34and do these experiments
7:36maybe throw them away maybe bring them
7:38back but at any point in time go back
7:40and look at the history of how this
7:41project evolved and then obviously we
7:44have a mailing lists and chats which are
7:46kind of a natural extension of how we
7:48would communicate but just take into the
7:50electronic form so the tools that
7:52open-source communities opt into our
7:54electronic they're also inherently
7:58asynchronous there's no means in an
8:02open-source community for me to demand
8:04the attention of somebody else right now
8:06I can ping them on chat I can
8:09you know send them an email but there's
8:11no guarantee that that message will come
8:13back to me and so there's no point in me
8:15necessarily waiting for that and I mean
8:18this is obviously open-source developers
8:19are rarely in the same place at the same
8:21time let alone different places at the
8:23same at the same time so these are
8:25constraints that they just have to
8:27accept and have to deal with this also
8:31means that the medium is locked free if
8:34I assume that I'm not going to get a
8:36response then I can't design processes
8:38that expect an immediate response and so
8:41this is kind of it you know if you look
8:43at like version control this is this is
8:44kind of an inherent design of
8:47distributed version controls through
8:49version control systems like anybody can
8:51start to move a feature forward without
8:53any dependence or reliance on anyone
8:55else you know I can check out a branch
8:57and start working on it you could check
8:58out a branch start working on it at some
9:00point we have to resolve that conflict
9:01but the progress of both of our work is
9:04not locked in any way you also in this
9:10in this process of open source you push
9:12approval and rejection to a review phase
9:15you don't have this a situation where
9:17you do a bunch of work do a bunch of
9:18work do a bunch of work and then finally
9:20ask for review once you're done like you
9:22can start that process as soon as
9:23possible and then the point at which the
9:27work is done you can continue on so then
9:34the nature of all of these tools
9:35basically means that the message is
9:36archived issues in pull requests show
9:40history version control obviously has a
9:43history mailing lists have archives an
9:46end chat has a log so you can see it at
9:50any at any point when you join the
9:51project you now can see the entire
9:53history of the project any time you have
9:55an issue so let's say you run into a bug
9:57in production as soon as you find out
9:59where that bug is coming from you can
10:02trace back the history of how that came
10:04to be you can look at the decision
10:05making process decision making process
10:07and all of that and hopefully Institute
10:10changes to prevent that from happening
10:12so well like here's a recent example of
10:16this I had a bunch of data a few weeks
10:18ago that I wanted
10:19figure out how to process and I'm like
10:21well wonder if anyone at github has ever
10:23used to do because I have I haven't
10:26worked on a team that has and so I go to
10:28github and I searched OOP and then at
10:30github which limits it to the github
10:32organization and I find out that there's
10:35actually three hundred and thirty-some
10:37conversations issues pull requests
10:39dealing with Hadoop and aren't one of
10:41you know Aria analytics team basically
10:43has a Hadoop cluster and so I was able
10:46to go ask them some of these questions
10:48so these are kind of inherently the the
10:51things that we get from the tools that
10:53we choose an open source so to review
11:00like here's here's the structure of
11:01communication and open source that you
11:03know the medium is electronic it's
11:05asynchronous it's locked free and and
11:08the result is that the message is
11:09archived so what does this have to say
11:12what does this have to do with
11:13transparency well here's here's the
11:16results of those those properties of the
11:18way that we communicate an open source
11:20information is exposed to others working
11:23towards shared goals simply the process
11:26of doing work makes that information
11:28available to them work towards the goal
11:30is rarely blocked if ever and anyone can
11:34participate regardless of geography
11:36timezone culture or even role so so
11:40whether I'm you know at a meeting or out
11:43to lunch or on vacation as soon as I get
11:47back I can catch up on everything that's
11:48happened in my project you know I didn't
11:52have to worry about missing out on
11:53information because a meeting happened
11:56and I wasn't there so these trans
11:59actually these transparent parent
12:02communication tools help to break down
12:04silos in our organizations and eliminate
12:07some of the tribal knowledge and so
12:09that's where we get transparency from so
12:14next I want to talk about participation
12:16so when we embrace these tools that are
12:18inherently transparent that create
12:20message archives as we work we naturally
12:25leave this paper trail right where
12:27anybody can come along and look at the
12:28work that we're doing and gain the
12:29necessary context but making this
12:32massive
12:33stack of information available is not
12:35actually enough to get people to
12:37participate in the work now that's being
12:39done so I want to talk about two things
12:41that I think are extremely important in
12:43the open-source world that we do that
12:46enables people to then participate so we
12:48work on minimizing friction and we work
12:51on automating review so my colleague Ben
12:56Volta that I mentioned defines friction
12:59in this way and I absolutely love it
13:00friction is the amount of time that
13:01elapses between I want to contribute and
13:04I have contributed so our goal should be
13:08to make that as close to zero as
13:09possible you know potential contributors
13:13should never have to come talk to us or
13:15ask us for information in order to get
13:17involved in a project so the way that we
13:20do that in open-source is we distill
13:21this huge body of information that we
13:24created naturally from the tools that we
13:26use so even you know basically it's not
13:30practical for somebody to come along and
13:31say okay I'll just look through all of
13:33the issues and discussions and archive
13:35and whatever and figure out how this
13:37project works until we work on
13:39documentation right which is as
13:41developers kind of our favorite word but
13:44the documentation is extremely important
13:46because it allows us to proactively
13:48communicate the latest thinking about
13:50how this project works again it doesn't
13:52matter what our methodology is but you
13:54take all of the information that's
13:55accumulated over time and you just still
13:58it down to knowledge that you can then
13:59give to anybody that's coming to the
14:02project so we do that with you know the
14:04readme is an extremely important tool
14:06and giving people insight into how to
14:07get started with the project and use it
14:08and then the contributing documents are
14:11you know how do how do you actually give
14:13back to this project now just how do you
14:15use it we also do that by automating
14:21something so to reduce friction every
14:24project inside of the github
14:26organization that has executable code in
14:28it generally has these three scripts so
14:31we give you script boot scrap script
14:33server and script test and I know often
14:35open-source projects will do something
14:36similar basically regardless of the
14:38technology that's written in whether
14:40it's you know a node project or a ruby
14:41project or even a c project you can
14:45you can clone the repo run script'
14:46bootstrap which will set up your
14:48environment you run script server or the
14:50equivalent to run whatever the code is
14:52and script tests will will run the tests
14:55so this allows you know like I'm not so
14:57extremely familiar with development node
14:59but I can clone any of the projects that
15:02we have at github that are node get them
15:05up and running make a few changes
15:07because I do know JavaScript quite well
15:09I can run the tests to make sure I
15:10didn't break anything and then I can
15:11submit a pull request so removing the
15:17friction of getting started is one of
15:19the most important steps to that that's
15:21generally where you lose people and
15:22unfortunately an open source project you
15:24don't actually know that you're losing
15:25them because they haven't talked to you
15:27at all right if our goal is to make it
15:28so that they can they can participate
15:30without asking permission then we don't
15:33know that they're getting stuck in the
15:34first process which is actually getting
15:36the thing running so the other way we
15:39enable people spay is we automate review
15:42and most of us have experienced this in
15:44like automated tests you know we write
15:47unit tests to make sure that the
15:48software does what we hope it should do
15:50and that it does what we wanted to do
15:52you know in six months or a year whereas
15:54people are changing the code but I think
15:58it's important to think outside of just
15:59the the realm of like traditional unit
16:01testing automate anything that you can
16:04anything that you do over and over and
16:06over again so like one of my favorite
16:09parts about this is that if you have
16:12really you know if you have things that
16:14you care a lot about it gets really
16:16annoying when you're constantly telling
16:18your your co-workers to also care about
16:20it but but you can automate it and put
16:24it you know in a script and then the
16:27continuous integration server tells them
16:28or Hugh bot if you're running that and
16:30now they're not mad at you they're mad
16:32at the robot so so here's an example of
16:38this we have a blog so to post on the
16:42github blog you open a pull request in a
16:45blog repo and since 2008 we've actually
16:48had 198 different employees publish a
16:51blog post and there's nobody inside of
16:54the organization that actually owns the
16:56blog
16:57it might belong to marketing it might
16:59belong to PR I think we're figuring that
17:01out right now but there's tons and tons
17:03of information over that we've gathered
17:06over the years about what we think is
17:07the right way to make blog posts and so
17:11one way to know we could have put that
17:12written that down in documentation said
17:14here's what we think is the best way to
17:16write blog post but instead what we've
17:18done is we've put that into code so
17:19every time we decide here's how we want
17:21our blog to work we'll write tests for
17:23it
17:23so when you when you make a pull request
17:25to the github blog there's at least you
17:27know these four tests now run the first
17:30one is is the the editorial calendar so
17:34at some point we decided that we want to
17:36only do one post a day maxximum and and
17:40when this became a problem it would have
17:41been I think a traditional organization
17:43would have been really easy to say oh
17:44like we want to prevent multiple posts
17:46from going out then we'll you know sign
17:48Susan to review every blog post and make
17:51sure that there's not one on every day
17:52but Susan probably has better things to
17:54do with our time so we can write code to
17:56do this so all we do now is the first
17:58person that opens up a PR gets a pull
18:01request gets to pick the day that they
18:02want their blog post to go out and now
18:04anybody else that opens a blog post
18:06it'll so it'll give them this warning it
18:08looks like you know somebody else
18:09already has that day so we've done this
18:12with a ton of other things we review for
18:16like certain grammar things there's this
18:18awesome node package called write good
18:20which my wife felt the need for me to
18:23explain to you that we know that write
18:24good is not how you would say you say
18:26write well but oh so yeah we basically
18:30just check for basic you know like best
18:32practices in writing we try to use we
18:34try to use active voice as much as
18:36possible or even things like a post
18:39should contain more use than wheeze or
18:41all images should be hosted on github if
18:43you go back to really old blog posts on
18:45github you'll see that some of them are
18:46missing images and so the point at which
18:48we realize that we're like oh let's add
18:50an automated test that make sure that
18:51every image we use in a blog post is is
18:54in a medium that we control so that
18:56it'll be here in you know five or ten
18:58years so the interesting thing about
19:03doing this as you you know so you use
19:05tools that create transparency you then
19:08start to distill that information
19:10into knowledge like as you automate it
19:14it now enables the people that you're
19:16working with to focus on substance no
19:18your when somebody opens a pull request
19:19you have to look like oh is this is this
19:22conflicting with another day are they
19:24using active voice all of those things
19:27now we know are taken care of and the
19:29people that are reviewing it can focus
19:30on is this good content is this
19:33something that we actually want to be
19:35talking about are we missing details you
19:38know that might be relevant to a certain
19:39audience so this is the value of
19:42automation as people as more and more
19:45people start to participate you get a
19:47focus on on the review of the substance
19:49so to sum up this participation to
19:54document the knowledge in prose or
19:56preferably code if you can if you do it
19:59in code then you can execute it for free
20:01without having to worry about the time
20:04that it takes so the last thing I want
20:09to talk is collaboration when we adopt
20:13these communication structures that
20:15expose information and enables people to
20:18participate then collaboration naturally
20:21starts to happen it's not something that
20:23we have to actually do anything about
20:25it's the side effect of the tools that
20:28we've chosen actually exposes the
20:31process that our teams go through so I
20:34love this about github like or any open
20:36source project really if you come in and
20:38watch the project for even just a day
20:40you start to see the way that the
20:42project works you start to see the
20:44patterns that they prefer and so you
20:47know whether it's your first day of the
20:48job and a company that does this or your
20:50first day an open-source project you're
20:52immediately start learning Jesse Newland
20:57who does a lot of chat ops at github
20:59works on our ops team I like the way
21:03that he says this you learned by lurking
21:04and you teach by doing which i think is
21:07a really interesting concept it's like
21:10just simply by doing your job you're
21:12teaching the person next to you to do
21:14their job as well and and again it
21:16doesn't matter what your process is it
21:18doesn't matter what methodology you use
21:20this process naturally expose
21:22or these tools naturally expose the
21:24process that you use and to quote Jessie
21:28again by placing tools directly in the
21:29middle of the conversation
21:30everyone is pairing all the time I think
21:33that's an awesome of concept so in
21:38summary these are the principles of open
21:40source we have transparency which
21:42enables participation which enables
21:44collaboration basically I mean if you if
21:49you take one thing away from this talk
21:51this is the idea prefer tools that
21:52inherently capture information and
21:54expose the process of your team and that
21:56might look different for your team than
21:58it does you know for someone else's team
21:59and so you have to experiment with all
22:02of these things and figure out what are
22:03those tools going back to Conway's law a
22:09co-worker of ours Jesse Toth gave a talk
22:13a long time ago where she finished with
22:15kind of this idea flipping Conway's law
22:16on its head and so instead of letting
22:19the organizational structure dictate the
22:21design that we choose or the design that
22:23we implement we should model our teams
22:24and our communication structures after
22:26the architecture that we want and what
22:28I'd love about this idea is that it
22:31doesn't matter what your role is in the
22:32organization like as you start to wit
22:35change the way that you communicate you
22:37can start to influence the structure of
22:39the organization so you know say you're
22:41working on a team that is very meeting
22:44centric and information is lost all the
22:46time your job on that team can be to
22:49start to document those meetings like
22:51create an archive find figure out ways
22:54to create that that communication trail
22:56and then maybe you know someday someday
22:58somebody's gone and misses a meeting now
23:01your notes become valuable and they can
23:02start to see the value of that or start
23:04to write some automated tests for things
23:06that annoy you and then someday people
23:09over time people will start to see the
23:11value of that hopefully slowly the
23:12communication and an team structure will
23:14start to change so these are just a few
23:18of the lessons I think we can learn from
23:19open source I mean I focus mostly on
23:21communication structure but I think open
23:23source has a lot to say about you know
23:25technical aspects like we can learn
23:26about good and bad design patterns we
23:29can learn about architecture we can
23:30learn about new tools I mean things like
23:32git itself came out of the open-source
23:34community
23:35I'm trying to solve challenges that
23:37those those people are facing we can
23:39also learn about social structures we
23:42can learn about conflict resolution
23:43overcoming cultural differences
23:46collaboration and coordination
23:48now there's economic factors so dealing
23:50with motivation resource allocation
23:53which is an interesting one in companies
23:55I hope somebody soon gives a talk about
23:58that and we can deal about political
24:00structures so what does it look like and
24:01for governance or decision-making within
24:03a company if you want to think be you
24:08know spend more time thinking about some
24:09of these things are reading about them I
24:10mean obviously the same advice that
24:12applies to open source can apply to our
24:14companies and so I'd recommend these two
24:16books the success of open source is
24:18written by a political scientist not a
24:20software developer at all but trying to
24:22understand what is unique about the
24:25software process the open source process
24:27and then producing open source software
24:29is a really practical book and actually
24:30how to do it how to do open source from
24:32you know picking an issue tracker and
24:35version control system all the way to
24:37dealing with conflict and and legal
24:39issues so I just wrap up I want to use a
24:44quote from the success of open source so
24:46the steam engine was the metal behind
24:47the first Industrial Revolution but the
24:49revolution was a set of ideas about
24:51organizing factories limited liability
24:53corporations trade unions and daily
24:55newspapers there's a lot of things that
24:58we can learn practically from open
24:59source but I think the bigger impact is
25:01going to be taking the process and
25:04applying it to other things
25:05I mean we're learning now how to apply
25:06it to our companies even though the
25:09source code doesn't end up being open
25:10other industries are learning how to
25:12apply it so open government open data
25:14it's going to be interesting to watch
25:16the trickle of some of these principles
25:18into these other organization or other
25:20industries so thank you very much
25:24you