More Buzzwords Won't Help: The Long History of DevOps Failures • Andrew Clay Shafer • GOTO 2023

[Music]foreignhow's everyone doingI think I'm between you and lunch so Ithink that's probably good because we goto sleep right after lunchso I make whatever kind of clickbaitytitle more buzzwords won't helpand then the subtitle was the long

history of devops failureswhich devops is something I feelsomewhat credible speaking about forwhatever reason let's talk about thatbut you could actually replace this withpretty much any buzzwordit's a long history of failures forwhatever whatever your favorite buzzwordis so if you want to substitutemicroservices continuous deliveryobservability like you can pretty muchdo that throughout the talkand then uh I'm gonna actually get intothe real history of social TechnicalSystems so we're going to see a little

bit of that and it should be fundramatic reading from papers that areolder than mebut firstwhen I talk about failure I thinkthere's two different types of failurethat are worth kind of noting so one isthe failure in an organization to getthe the results that they hope to getwith their investment and then I thinkthere's another kind of macro failurewhere the the movement or the kind of

buzzword itself didn't didn't accomplishsome of the things that it wanted to inin the moment and across time there'sdifferent kinds of failures at thosescalesand this is one of my favorite sayingsdoes anyone ever study Physicsso so uh does anyone know what planesconstant isso there's this there's this period inphysics whereessentially everyone thinks that all weneed is more precisionand and Newtonian mechanics all the way

to the bottom like it we got it allfigured out we're just going to get moreand more precise when we're gonna figureit out and they start measuring thestuff and it getsto expose these anomalies about how thethe fabric of reality actually works butit turns out that you can't just show upwith data and say like this is the wayit answers itself we can't explainbecause the people who build theircareerson the old Paradigm will basicallynot allow the new paradigm to come inlike birthing new scientific paradigmsis a messy process

and so this guy blinks principle is thatscience advances advances one funeral ata timethat the new ideas don't come and andhave the gravity of their own weight andMerit but that the people who wouldbenefit from the old order slowly phaseout and the people who understand thereality of the new paradigm are the onesthat are in chargeso for every complex problem there'sanswer that is clearsimpleand wrong

another great quoteand then I'm gonna try to bring thisback at the very very end but there'sthis there's a story about scurvywhich is a disease that you get if youdon't get any vitamin Cbecause vitamin C is critical forbasically collagen which is whatstitches your whole body togetherand there's evidence that Citrus it wasbasically written down with experimentsand have shown that Citrus will preventscurvy but it didn't really get

implemented for over a hundred years onsome of the ships so this is kind oflike it interesting there's a fewdifferent versions and people debatesome of the accuracy of a uh for forlike all the timeline and all the actorsbut for a long stretch of time there'sthis thing that exists that you canpoint to that says if you eat citrusfruit you won't get scurvy and it takesa long time before people stop havingscurvy on shipsand then this other thing will bringback a little bit and this is just kindof Representative of at least for mewhen I see stuff like this

I wonder what these people knew like Ifeel like they knew something right tobe able to build these kind of thingsthese artifacts that stand is atestament to some some knowledge someunderstandingso I'm going to introduce myself so thisis a little bit of a a story that someof you might have suffered throughbeforewe'll do the short version sothis is me I spent the last we'll sayI don't know 20 years working on opensource automation Cloud automation

projects from Puppet to openstack toCloud Foundry to kubernetes I wrote somebooks I did some conferences and had agood time doing all that made a littlebit of money it was really great devopsis great for my bank account and I juststarted this new company ergonautics sowe're gonna have some Adventures if youlike some of the stuff I talked abouttoday have you come talk about it moreand then I come in variousconfigurations of hair and beard whichis critical for the next time you see mebecause I probably won't look the samelive in ArizonaTwitter used to be a thing that I liked

I don't even know what to do with it nowum but yeah so here's here's a bunch ofRepresentative configurations of hairand beard so next time don't be afraidit's the same same monsterso moving on from thatI invented devopsyaythis is embarrassingbecause sometimes people actually saythat or introduce me like that I'm like

I did not invent devopsI stole itso I was very lucky with you know someof their friends and some of my kind ofpersonal choices and circumstances towatch a bunch of things happen in abunch of places and I put together likemy own version of that and I was I don'tknow loud and obnoxious in the rightplace at the right time for people tosay oh here's this guy that said thesethings so this advice this is this isone of the actual bits of advice from

today the good good devops copy stuffbut great devops steal so wherever youcan find a lesson you should use itand over time I think it's funny becausethere's there's all these conversationsabout devops isn't defined enough devopsif if it was only better defined thenthen like we'd we'd be able to do itbetter and then the marketing peopleruined it or whatever and then and thenthen SRE but comes along and it's likeall very very detailed and everyone'slike yeah we're sres okay well do you

have slos no do you do prrs no okay whatdo you actually do we're just like CISadmins before but now we have a newtitle so it's like okay I don't believeyou I don't believe that definition willhelp you but this is mineso for me and this is going to be arecurring theme this like a socialtechnical system stuff optimizing TheHuman Experience and performance ofoperating software with software andwith humans so it's just the extensionof software's eating the world we'regoing to use software to make softwareto operate software blah blah

and I didn't really want a new word likewhen devops started being a wordwhat I was talking about at the time andwhat became kind of like the primordialdevops conversations was agileinfrastructure so my my comeuppancewhatever is really focus on softwareengineering and I wanted to do Agilestuff and not not like scrum stand upstuff but like XP buy the book CI and Iwas very frustrated working for astartup that was a they had an

e-commerce platform and I kept having tostay up or or worse uh going to bed andthen finding out at 6 a.m when the whenthe East Coast couldn't do anytransactions that we broke the systemright so the the differences betweendevelopment testing and production Iwanted to make that reproducible buildfrom Source extend to my infrastructurebecause I got into this mindset where Ibelieved in the religion that if you cankind of like tdd stuff then you havemore confidence and I just want toextend that into my infrastructureso now what I we're against buzzword but

we need all of them right so this iswhat we're trying to do right now thisis what we've been trying to do for awhile continuously devops microserverlessthey'll fix everythingwith software and with humansthis is how I feel listen to a lot ofpeople talk about devops or write blogposts about stuffum and then now there's this new wave ofstuff about how devops is dead which ispretty funny since it dies every yearfor the last like 10 years

uh it's usually set up as sort of likethe straw man argument about you buildit you run it which is a quote from 2006which is three years before Dallas iswearing but whatever we'll come back tothat so my my personal idea is aboutdevops agile whatever we're deeplyinfluenced by this guy uh W Edward Demiand this is a quote from the book TheDemi wrote It's it says one gets a goodrating for fighting a fire the result isvisible can be Quantified if you do it

right the first time you're invisibleyou satisfied the requirements that isyour job mess it up and correct it lateryou become a heroand that's a very good talk that wasgiven at a devops Days in 2010 and thenI watched that same talk again at theSRE conum in last October so it's like the sameidea is kind of built on top of eachother about hero culture and qualityit's like this the principles areeternalso during the second World War

gaming worked as a quality expert in ourown plants throughout the United Statesand and he did a bunch of work and hisapproach the the quality process that heimplemented based on statistics andmetrics all this stuff and andcoordinating across the the workflowshelp produce higher quality products uhraise the productivity and lower thecost like demonstrable data driven thingokayandthat that was an approach where he wasthinking about as a system right so it's

a system and he's trying to improve ithe's measuring he's trying to define theprocess describe the process and make itbetter like let's make it better andthen the war endedand then all the soldiers came back fromthe war and they're like that's finewhen the women are running the factorybut we're not doing thatso he had all the data but no one wouldlisten to Demingin the U.Sand we'll come back to that so thatmight be the end of itmeanwhile

in the 1950s British coal minesso this is where this is where the termsocio-technical first comes into lexiconthis is the research that comes in sothere's this guy trysthe's at the Tavistock Institute and he'sstudying coal mining and at this time inthe 50s what's happening so there's thisindustrialization and and investment inAutomation in the coal minesand the the heavily automated coal minesare seeing lower productivity uh lower

safety and lower moreland Tavistock or Tris is studying thesekind of methodologiesand he makes a few observations some ofwhich might sound similar to others soI'm just going to do a dramatic readingI probably won't read all of it I have abunch of it so I'm I'm gonna I just wantyou to understand like this is the 1950sright so this is like 70 years ago thisis before I was born and I'm oldthe work organization of the new scenewas to us a novel phenomena consisting

of a set of relatively autonomous groupsinterchanging roles and shifts andregulating their affairs with theminimum of supervisioncooperation between task groups waseverywhere in evidence personalcommitment was obvious absenteeism lowaccidents infrequent productivity Highlike it's the stuff this is the stuffright thereand and this is it from anotherparagraph a little later theorganizational model that fused Weber'sdescription of bureaucracy with

Frederick Taylor's concept of ScientificManagement had become pervasive Thehaymore Innovation showed us there wasan alternative so right nowI don't know who he listens to but ifyou go listen to enough people you'regoing to hear people who say like Taylorisn't bad this is Taylor is as bad in1950 rightso the social technical systems theorykind of the defining quality of it isthat an organizational system can onlybe understood and improved if the socialand Technical are both considered as theindependent parts of a single system

that if you just focus on the technologythen you won't get a good outcome or asgood of outcome as you could if youcalmed it and think about the socialside of it too and they they have to bein balance and if you go read theliterature you're going to run into thisword joint optimization or this phrasejoint optimization all the time abouthow you have to have this balancebetween the social system and thetechnical systemso some of the principles involved wereas follows again this is the stuff fromthe 50s the work system which comprise aset of activities that made up of

function hold now became the basic unitrather than the single jobs into whichit was decomposableso it's a cross-functional teamcorrespondingly the worker becameCentral rather than the individual jobholder so it's about that that workbeing produced by the unit of people noteach person doing their jobinternal regulation of the system by thegroup was thus rendered possible becausethe external regulation rather than theexternal regulation by individuals bysupervisors so we don't need managerswe're going to be this autonomous self

thing yeah has anyone ever heardanything like thatin any of these kind of conferences inthe last you know we treat like it's newrightthe design principle based on theredundancy of functions rather than theredundancy of Parts characterize theunderlying organizational philosophythat tended to develop multiple skillsin the individual and immensely increasethe response to a patrol of the grouprepertoire of the group so this is againthis Echoes of all this stuff that Ihear people saying about softwaredevelopment right now you know the the

full stack developer being able to doall these things across the thedifferent things so this principlevalues the discretionary rather thandescribe part of work roles there's alonger I won't give today about howevery organization is held together bypeople who do things that aren't theirjob whether you know going to the fastfood restaurant or the most technicalorganization you can imagine if everyonejust did their job everything will fallapart guaranteed so it treats theindividual as complementary to themachine rather than an extension of itand last it was variety increasing for

both the individual and the organizationrather than variety decreasing in thebureaucratic mode so you're trying tohelp these people be able to do morethings as peopleI'm not gonna read this whole thing butso this is basically tristafter the fact lamenting the idea thatlike once you've seen this like youcan't go backright the idea of these separateapproaches the social and TechnicalSystems would no longer sufficefor once such as myself we'd experiencethe profound consequences right so he's

seen this thing and he he has all thedata it shows it's better productivitybetter safety better moraleand then what happens oh this is a goodoneso this again I don't necessarily needto read all these you can read themyourselves but this is the old paradigmwould sound suspiciously like how loveEnterprises work nowand then this is the new paradigm whichsounds like what everyone's trying tosay from devops Dave going back 10 yearslater 20 years almost now 15 years sothe technological imperative like we're

going to be better technologists the thenew paradigm is that joint optimizationman also will forgive them for theirgenderum language or whatever this is adifferent time so man is an extension ofthe machine man is a compliment to themachine Matt is an Expendable part sparepart man as a resource to be developedand on and on and on and then last butnot leastlow risk takingInnovation with everyone seems to wantright that's what everyone saysinnovationI have another round that Innovation isnot what you need you need the most

obvious solution to your most obviousproblem but that's we'll come back tothat in another talkso the interest in the Haymarket so sowe have all this data we're seeing thisnew paradigm like what do you think isgoing to happen right everyone's goingto just go adopt thislike we have the data we did the worklike we're just gonna everyone's gonnaIt's Gonna Change coal mining we gotthis we're going to do this better rightthat's where that's what's going tohappenit's obviousso that's not what happenedso so right here in the middle we havethis report that shows all this stuff is

better and the divisional board howeverdid not wish attention drawn to itthey feared the power change that wouldbe consequent on allowing groups tobecome more autonomous at a time whenthey themselves were intent onintensifying managerial controls inorder to accelerate the fullmechanization of the minesso the data shows that this fullmechanization is lower productivitylower safety lower morale for likethat's what we're doingso then this goes even farther and they

they have reached the wholeorganizational level or researchedthrough the efforts show that this isbetter it gets to the divisionalchairman and he's like yeah we're notdoing thatandthen the the end is um the sad part uhthe union regionally negotiated specialpay for operators of the new equipmentso if you use that new tool you get likebetter pay this broke up the unity ofthe face groups which were furtherdecimated when bonuses were introducedfor various classes workers within a

year or two the conventional systemreinstated itselfyayand then a report was submitted to theNational coal board the results were notdisputed no no one questioned the databut the warrants priorities wereelsewhere and then you kind of can skipto the middle right here it saysthey stopped they basically buried itbecause the the energy markets wereshifting to oiland so the argument's like oh that's foroil or that's for coal but like now

there's going to be oil so like we don'tneed to worry about thatso they yeah so that yeah sadand then here's another thing I think isinteresting because you see all thesepeople trying to do like autonomousgroups and all this and it doesn'talways work out I don't know if you everif you've ever been a new manager andyou thought I'm going to make everyoneempowered and then they all Panicum I tried that before so autonomousgroups not always succeed a good dealhas been known about the conditionaffecting their success or failure thesewill now be reviewed here except to notethat one of the most common reasons for

failure is lack of support in thesurrounding organizationalanyone ever seen thatso this this is just on and on and onthe same kind of stuff so very earlythat they showed that thisthe evidence is there but the jointoptimization involves differentprinciples from from following thetechnological imperative so if you don'trecognize the social system it's hard togo back and kind of like redo it afteryou've already tried to mechanizeeverything

and I just feel like when you readpapers like this it reminds me of likethey knew somethingand then and then we just didn't get itso now we're going back to Demi so theprimary reason we know of Deming isbecause he went to Japan so he goes toJapan who's just beenumBond and has to rebuild all this stuffand he goes there in the 50sno one really knows about him until the80s so there's a documentary that's madewith this title you can actually go

watch the whole thing on YouTube IfJapan can why can't weand Deming says and there's a quote hereuh I think that people here ExpectMiracles American management thinks theycan just copy from Japan but they don'tknow what to copyand I think that sort of mirrors a lotof what I see organizations trying tocopy like Fang or whateverum some someone gave a conference talkum this guy was ignored no one knewabout him until the 80s comes backand if you don't have Deming you don't

have Toyota you don't have leanlike all this stuff people talk aboutlean came really from himso when American Executives visitedthese Japanese factoriesthey thought the Japanese were lyingthey thought they were hiding the truthbecause they they really couldn'tbelieve that they were capable of thisjust-in-time Manufacturingwhich to them probably sounded similarto a conference talk when uh someone

said I deploy 10 times a dayit's like there's no way you do thatit's irresponsible like they thoughtthey were hidingthey thought they were hiding inventorybecause they couldn't conceive of aworld where there wasn't inventory forthem to run the way they ran in the USso when they see the factories they theysee the robots and they know they'retaking the market like they see the thenumbers and they see their cars on theroad but they don't believe they'redoing this thing so what what is theAmerican Auto industry dothey they see what Japan's been doing

they don't believe what Japan tells themthey're doingand they decide to investin automationso the this is the headline from thisarticle which we can go pick up I'llsend your link if you want GMC clienttruly began with his quest to turnpeople into machinesand the the punch line is reallyat the bit at the middle so they spentall this money on on automation tryingto make their factories automated andthey spent 45 billion dollars and 45

billion dollars in 1980 is a lot morethan this today still a lot for for alot of things uh so they should theyspent 45 billion dollars despite thatspending its global market share Rose bybut a single percentage point to 22percent for the same amount of money wecould buy Toyota and Nissan outrightssaid the executive which would haveinstantly bumped GM's market share to 40percentso go buy that tool it's going to befineand I'm just going to dig in a little

bit more to the GM stuff so quality hassufferedbecause instead of making Flawless carsworkers simply did their assigned jobsworkers had no big picture goal ofbuilding cars together to motivate themso if you understand total productionsystem one of the things that everyonehas at least in the you know version thethe fantasy version that you get fromthe book is they can stop the linewhich was Unthinkable in in the American

formulation of this because it's like wegot to be productive you got to beproductive anything that's idle is wastebut in in the bigger picture that Japanhad everyone understood as part of thesystem the goal is not to maketo do my job my goal is not to do my jobmy goal is to make sure that the carthat gets built is high qualityrightit's not my it's not just my job it'severyone's job to deliver these high

quality carsso focusing on the technical withoutaccounting for this socio never it goesvery wellis anyone doing thisyeah have you ever seen thisall you do is automate confusion and Igot in a lot ofI don't know over time my ownunderstanding evolved right because Iused to be automate all the things yeahwe're gonna do this and then you do thata few times and you realize all you didwas automate all the bad stuff rightlike we just made it what we really need

to do is think about why we do thingsand what we really need to do is takeadvantage of and then and then that'sbefore you get to the the architecturesthemselves and the way that the humansinteract with those architectures haveimplications and that if you automatesomething that wasn't designed to beautomated who who deals with things ourLegacy systems are a lot older than like10 years 20 years right so like when yougo try to automate those with the newtools you're going to have an extra badtimeand it's worth revisiting thearchitecture as much as it is trying to

automate itand this is a I think this is a paperthat should be required reading foranyone who wants to work on computershas anyone ever read our news ofautomation or seen thisyou see it thrown around a lot of devopsconferences or if you follow John allSquall you'll you'll the price like it'sit's mean time to the ironies ofautomation as a as a metricbut there's a paper that walks throughand I can't remember the exact Dave it'slike 80 somethingum but it walks through how when youautomate things it doesn't mean that you

need less humans or maybe you need lesshumans for the certain thing but ifhumans are not involved there's no humanfactors then then the the failure modesare bad right so there's a bunch ofthings about automating uh aircraftnavigation automating these factoriesand it's it's not that you can't getmore from the machines it's that if youdon't have some understanding when thoseabstractions have got so far away from

the reality and and there's someunderlying assumption about theabstraction that's broken then thehumans have a very hard time dealingwith that and there's a bunch of there'sa bunch of great examples of thatso the more advanced a control system isthe more crucial may be the contributionof the human operator so that's a directquote from the paperthe more the more advanced ourautomation gets the more critical thesocial side will beand you know we'll see we'll see howChad GPD does over the next uh

what you what we're gonna have like uhchat on chat violence coming out of thisvery soon butit's fineso despite anyone's hopes to the countrylike I think that a high level there'sthere's people who believe or or hopethat we could just you know we'll we'llautomate it the machines will do it andwe won't have to pay peopleright because if you go down this wholetoo too deep you start critiquingcapitalism but we're not going to dothat todayyeah but we're gonna get to this so this

iswhere everyone wants so if this was easyeveryone would be good at it right nowright if devops was easy whatever it wasbuzzword right so SRE platformengineering it's like all the same stuffso Aaron wants devopshere's what here's what mostorganizations wantwhat they really want what they reallyreally wantreliabilityavailability

scalabilityoperability usability observabilityand probably some realities and yet itisall for freewithout changing anythingand then just to make that point II just like to makeattacks really bigthat's that's what most organizationswant

another thing you have to be careful ofas you're engineering your socialtechnical system is that performancealmost always goes downbefore going back upand you have this littleJ curve of change is hard problem thatyou have to solve so that you initiatesome change that you have a certainlevel of performance and there's peoplethat will initially resist that forwhatever you know reason whatever theirincentives are and then

if you if you don't have enough kind ofinertia and and support to get over thatbump and back to their side then you canend up with that being used against thechange initiative and and it willeradicate any of the hope that you youthought you're going to have with thechange right so you you have this lackof commitment to changeum teams teams are not given time tointegrate the new processor technologyyou didn't you didn't invest enough inthe social side so that the new teamscan use the tool that you got

and then but what you want to get to ishigh performing and hopefully you'rekind of like always going up but this isthe reality of change and it's easy tosee this as an individual if you want tolearn a new skill you want to be betterat shooting foul shots you want to bebetter playing a musical instrument youwant to be very chess when you try newthings you tend to go down yourperformance goes down and organizationsfor whatever reason don't don't alwaysaccommodate that political realityand then another thing that you see andthis is there's there's more researchbut I don't need to read more research

papers for you about uh this this kindof force for how organizationscome to look like each otheror how we like copy each other'spractices and this used to reallyfrustrate me in the early days doingkind of like agile stuff before it wasdevops and devops again and now watchingwhatever the buzzword is thatyou see this this tendency to take thevocabularyand change nothing elseand and for a while that frustrated me

but when you get into literature youunderstand a little bit more about whythat happens and the the bestexclamation or the simple version thatI'm going to give you today is that inthe Innovation and the early adopterstheir motivation is the advantagethey're seeking an advantage so they'rethey're trying to win some game thatthey're playing they understand the gamethey're playing and they go make themoves that they believe with their modelwill improve it and what happens then isyou getthe door metrics or you get something

that legitimizes that practiceand then as it moves into maybe thefront lip of the early majority has alittle bit of a Advantage seekingtendency but definitely by the time youget to the middle of this and the latemajority they're they'reorganizationally seeking legitimacyright so they want to use the vocabularybecause that's what the cool kids aredoing that's what was legitimized bythese metrics and that's what we we wantto be the cool kids too even though youknowor not so the words cross the chasm

before the understanding and thepracticesand I think you can see the waves ofthis if you've been involved in softwareagile whatever going back there'sthere's lots of organizations you showup and you're likeoh we're doing agileokay oh we're doing devopsall right it's like why are you so badat software thenlike literally you're terrible thesoftware well like oh we we havestand-ups and we stopped writingdocumentationokay cool did you see I no of course not

that would be hard testing's hardso new words cross because they're newright so there's there's a certain thingthat makes us excited about new thingsright like the analysts need new wordseven though it's the old thing eventhough it goes back to tryst in the 50sit's the same principles we gotta wegotta have like this new word to put onthe hype cycleand what you see over and over

if tech is more about fashion than it isabout Meritright like whatever's the cool thinglike we gotta do thatit's a bad idea the other thing thathappens which is depending on yourperspective good and bad is we we needto fund a new initiative and so we needa new word because the old word alreadyfailed right like we we had an agiletransformation and then we had a devopstransformation you knowwe gotta have some word because we knowthat other thing that we already diddidn't work

and then also tribalism because we gotto other each otherit's fun and related to that as a orgchart so people need finding an orchardthis this is something I say all thetime if you show me your org chart andyour funding model I'll predicteverything that's hard for you right nowso it's the new words but it's still thesame system and then people are shockedthat they have the same outcomesso very few organizations are activelyoptimizing their process

and I I like to think of that process asa I mean not to compare myself to gamingbut it's like you have this thing we canstart to think about and measure it andit's about your context it's not aboutcopying someone else's process and thisis related to someone that was askingquestions charity is very fewindividuals are empowered to reallychange your process in fact if you tryto change your process in a lot ofplaces it's it's you're going to have abad time and and yeah ask me how I knowso the best time to plant a tree was 20

years ago second best times nowfor gold legitimacyseek advantage that's my advice to youchange your organizationor change your organizationit's easy rightespecially now that everyone's laid offso we can't just copy we can't copy theNetflix you know all these other thingsright we can't do thatI mean we can it just won't be good

because we don't have the samepreconditions they do and we don't havethe same winning conditions that they doand we don't know what to copyso we we when when organizations copythey usually copy the most superficialthingright so people try to copy Amazon'sI don't know API but who who's tried tocopy Amazon's writing culture who'strying to copy like all these otherthings that they do organizationallyso there's a hard problemit was hard for Deming and it was hard

for tryst they had all this data theywere empowered and given this thing andactually produced resultsand then people didn't listen to themso don't feel bad if they all listen toyou right or meso parting thoughtshard problems are made hardthis isn't a technical problemthis isn't a people problem we have tosolve both these things as a singlesystem togetherdon't give up

so I I saw that picture as funny[Laughter]a bad system will be in a good personevery time it is not necessary to changesurvival is not mandatoryanother like 50 year old quoteso thanksthis is uh me Andrew ergonautic we focuson trying to help people build their

people practicing platformsand I give you permission to learnanythingfrom anyoneand I think we have about 15 minutesleft for questions[Applause]