“Agile Development” – The Software Industry’s New Meme

AgileMeme

“Agile” as A Meme

Since the introduction of Agile Development in the early 2000s it has increasingly spread throughout the software industry in the same manner a popular cult is formed; based on supposition, mythology, and rumor for which little real statistical proof has ever been provided.

A more accurate description of the growing popularity of this development paradigm is that of a sociological “meme”. Wikipedia defines the word “meme” using basically the same definition as found in the online version of the Merriam-Webster Dictionary, a standard in English studies, as the following…

“A meme is “an idea, behavior, or style that spreads from person to person within a culture”. A meme acts as a unit for carrying cultural ideas, symbols, or practices that can be transmitted from one mind to another through writing, speech, gestures, rituals, or other imitable phenomena with a mimicked theme. Supporters of the concept regard memes as cultural analogues to genes in that they self-replicate, mutate, and respond to selective pressures.”

The Wikipedia description goes on to elaborate on the transfer of memes in the same way that is found in biological evolution.

“Proponents theorize that memes are a viral phenomenon that may evolve by natural selection in a manner analogous to that of biological evolution. Memes do this through the processes of variation, mutation, competition, and inheritance, each of which influences a meme’s reproductive success. Memes spread through the behavior that they generate in their hosts. Memes that propagate less prolifically may become extinct, while others may survive, spread, and (for better or for worse) mutate. Memes that replicate most effectively enjoy more success, and some may replicate effectively even when they prove to be detrimental to the welfare of their hosts.

In other words, memes are transferred in a culture the same way that natural selection works in biology; except for the fact that with memes, which are transfers of information, they become successful not as a result of intelligent research and understanding by the Human transfer points but in somewhat the same manner that fads are often found to grow in popularity, through simple popularity of an accepted strain of thought.

Birth Pains

When “Extreme Programming (XP)” was developed prior to the Agile movement, it was defined in the fourth year of a five-year projected development timeline for the Chrysler C3 Payroll system, a system that was to be developed to replace the existing payroll processes. The people that developed the XP development paradigm thought they had found a “silver bullet” to software development with their dilution of planning and analytical design coupled with the new idea of “paired programming” mixed with a “Code First” strategy.  One would think that after working with such development concepts over the four years of their project they would have been actually on to something. Not waiting for the final results in the form of a completed and delivered project into production, the “Extreme Programming” creators went out on tour around the US to popularize what they believed they had developed as the answer for speeding up software development in a time when technical management was coming under increasing pressures to get their deliverables in to production faster. These promoters elicited a lot of interest and the basis for a new software development paradigm was born.

Unfortunately, in the 5th and final year of the project, the entire thing collapsed under the weight of these newly derived concepts proving that they were erroneous from the beginning.

Nonetheless, a new software industry “meme” had been born as a result of the popularization efforts by the creators of XP. This new meme would have found its way into extinction had it not been for the efforts of a new set of developers (with probably the same people who created XP) who came up with the “Agile Manifesto” promoting a more refined methodology where the best tenets (as they believed them to be) of XP development could be incorporated, while adding alternative and\or refined concepts to the “Agile” paradigm, which were seen as corrections to the original XP model. So with “Agile”, paired programming was an option and not a requirement but nonetheless still promoted. The “Code First” strategy was still followed while planning and design were diluted with the idea of collaborative development with the user community; another concept that has proven to have mixed results since many users really don’t care how their systems get built.

In fairness to the developers of both the XP and “Agile” techniques, there were two major, external forces that lent their influence to these developments.

First was the excited frenzy that surrounded the rise of the Internet as a commercial medium for transactional processing, which in turn gave rise to the “DotCom Bubble, where clueless venture capitalists and to some extent organized crime invested millions if not billions in an outrageous number of new start-up companies all organized around bringing new forms of consumerism to the Internet.

Such start-ups were under heavy pressures to develop new applications that would allow everyone to buy literally anything on the Internet from credit to physical goods. The “meme” born from XP made it’s way into these small firms who were all under the impression that they had found something completely new and unique with this new form of rapid application development paradigm. Combined with the inexperience of the new youth oriented culture of these new start-ups, the development techniques they employed against frivolous plans and goals that in many cases made little sense, all came together in the first software development business “bubble”, which eventually blew up in their faces pushing the newly minted “DotCom Era” into a sudden free-fall it never recovered from.

However, the idea of using the Internet for commercial processes was not as farfetched as the failure of so many companies made it appear to be. The Internet was certainly a viable medium to work with so everyone went back to the drawing boards to begin again.

Saved From Extinction

At the same time the outsourcing of US technical jobs was becoming as much a frenzy in the early to mid-2000s as corporations saw the Internet as a communication medium that could be used for connecting technical departments to foreign counterparts in countries such as India and China that could provide similar technical resources for far lower costs than what could be found in the United States; or so it was thought as no one gave any thought to the costs in administration, management, quality control, and project control to name a few ignored areas in this rush to cut the bottom line.

This rising inequity within the US corporate sphere gave new impetus to US technical personal to adopt techniques that rushed out deliverables at greater speeds trying to prove that US developers were as every bit as good as their lowly-paid foreign counterparts. The result was that instead of maintaining the original quality of the US software community it instead slowly adopted the “Agile” paradigm allowing corporations in turn (though they were doing it anyway) to reduce or eliminate vital components to the development of quality software such as business analysts, system analysts, software designers, architects, quality control centers and the like pushing increased responsibilities and pressures onto software developers and engineers all with the ridiculous idea that software development teams could do it all.

Under such circumstances the “meme” that was born with XP found a new nesting place in the developing Agile Community of proponents. The result was the growing contention that software developers could do everything while also adopting in-depth knowledge of the businesses they were working in. Article after article appeared in the field’s technical journals proposing as much and technical managers looking for ways to reduce costs and speed up the delivery of their products “drank the Cool-Aid” giving this nascent meme a new lease on life instead of the extinction it should have experienced.

Today, hardly a technical site ignores this as one article after the other is published proclaiming the benefits of Agile, Team-Agile for enterprises and the new paradigm of DevOps, which appears to increase responsibility by including operations personnel as part of the software development team as well as their responsibilities. The industry’s propaganda engines have been on full throttle for quite a while and the deterioration within the software development industry has demonstrated this. Even Microsoft has fallen prey to this infectious disease of high-speed development without proper structural planning and analysts have recently made note of the deterioration in their product quality after switching to the “Agile” paradigm.

Nonetheless, in all the years that this meme has infested the software profession very little hard-core analysis has been performed in the same fashion that long-term analysis of earlier software development efforts and the many impediments to quality product delivery had been analyzed. There are some articles pointing to the successes and failures of the Agile methodology but for the most part they are few and far in-between with little to no actual in-depth discussion has to how the projects were actually implemented. People who have written such articles that accurately describe Agile as a failure in their own experiences are merely told by proponents that it wasn’t implemented correctly or some facets of the paradigm were not in included.

What Did We All Do Before “Agile”?

However, prior to even XP, the “bible” of software development (“Rapid Development”), was published in 1996 by Stephen McConnell, which outlined every major aspect on how to develop good, durable software products without all of the hype of the paradigms today. This book is still so popular it is still available on Amazon.com in its original 1st edition publication. And prior to its publications there were many examples documented of how good software projects were structured. In one such case an entire book that included all of the framework source-code for a highly successful project for the state of South Dakota demonstrated how Visual Basic 6.0 was used for both the project’s desktop and web-based applications.   The concurrent usage of this system was over 1500 users! “Agile” was not included just standard, well-defined N-tiered development processes.

Instead of looking deeply into the past for what actually worked, creators of XP and Agile only took a short glance and found its only target being that of the “Waterfall Approach” to software development demonizing it as the worst possible way to create new software while promoting the new concepts as the saviors of the software industry. And if one were to apply the “Waterfall Approach” to ever type of software development project that accusation would be quite correct. However, the way Agile is being promoted is in fact the same way that the “Waterfall Approach” is being demonized since Agile promoters see it as the panacea for all types of projects, though they have never offered any substantial proof in order to make such a justification.

The reality is that the “Waterfall Approach” is merely one type of development paradigm, which has been quite successful for the large, complex projects that it was originally designed to be used against. However, there are approximately 10 different types of software development paradigms that have been in use for years that the current crop of software professionals appear to have either ignored or never bothered to research. If they had, their theories on “Agile” would fall apart. To demonstrate this, here are the standard development models that have all been used to date in terms of software engineering analysis…

  • Pure Waterfall
  • Code & Fix (what many organizations still use to their detriment for all projects)
  • Spiral
  • Modified Waterfall
  • Evolutionary prototyping
  • Staged Delivery
  • Evolutionary Delivery
  • Design to Schedule
  • Design to Tools
  • Off The Shelf Software Implementation

Each of the development models above have their strengths and weaknesses and each has to be researched for its appropriate use for a new project under development.

Surprisingly however, the “Code & Fix” model is what Agile today mostly mirrors, which is fine for simple maintenance tasks but useless for complex maintenance and complex project development. In this former vein, Agile has proven somewhat successful but has found its limitations when applied to these other larger projects.

Nonetheless, like “Code & Fix”, Agile is being used for many different types of projects as a result of the effectiveness of the belief that “Agile” can be applied just about anywhere in the software development spectrum (an underlying foundation to the meme contention). And being as embedded in the software industry’s culture as it is, “Agile Development” has become the model for all such development making it not only a “cultural meme” in the profession but a dangerous one similar to what happens with cults; unquestioned belief.

Questioning The Dogma

There are those who have questioned the promotion of the “Agile” model and have done so quite well, though such people are ignored in today’s conformist society since for most it is now too difficult to do any real critical thinking on matters within a culture that promotes sound-bite thinking.

Capers Jones, a highly respected software engineering analyst, has an excellent paper that demolishes the concept of paired-programming, which can be found at the following link… http://namcookanalytics.com/high-costs-and-negative-value-of-pair-programming/

In his paper, Jones demonstrates the faulty economics that paired-programming forces on technical organizations as well as demonstrates statistical analysis showing the failures in such a development technique.

In another well written paper by Lajos Moczar and published in CIO magazine, he demonstrates that “Agile” simply lacks any common sense in its approach to software development. The paper is short but concise in its presentation and can be found here… http://www.cio.com/article/2385322/agile-development/why-agile-isn-t-working–bringing-common-sense-to-agile-principles.html

Finally, in a third paper, written by Alex Papadimoulis, “Agile development” is compared to building a pyramid sideways. The comments are as interesting to read as the article as you will find both supporters of Alex’s contention as well as detractors. Admittedly, the concept is a little far-fetched but the author makes some very good points.

And not surprisingly, such papers and subsequent analysis today is rather difficult to come by as it took quite a while to dig these up in research. Instead, there should be a lot more out there with similar research and points of view but there isn’t as “Agile” thinking has literally taken over the psychology in the business application development corner of the software profession. And anyone who tends to disagree with this is basically told they don’t know what they are talking about.

This process has also shown the same deterioration in critical thinking in software development that has been shown in many aspects of US national sociology, especially education, as demonstrated in well documented, sociological studies. With mobile-computing taking center stage in most popularized software development today, one needs no longer the critical skills in complex system development that were once the hallmark of professional US software developers and engineers. Instead, mobile computing requires a rather limited of skills in order to build non-complex applications considering that only so much can fit on a smart-phone with only a proportional increase in complexity for tablet-based software. And yet, what is actually being developed for these devices?

In fact, if one were to take a look at the Windows App Store and its breakdown of stored applications (which has probably the same amount as the Android Store has), practically 90% of such applications are devoted to games and entertainment leaving one to wonder exactly how necessary such “junk consumerism” is to our national, psychological health.

“Agile”, which has done a rather good job in reducing software development processes to nothing more than sound-bites in terms of any real discussion that surrounds it has in fact seen a number of what appear to be well designed documents discussing various aspects of “Agile”. However, the graphics used in these documents all of which have shown nothing more than photographed scratches on paper for diagrams of the intended topic appears to demonstrate the rather casual approach to the serious subject of software development that this paradigm supports. One would think that the authors, if they were serious about their subject matter, would have created better graphics that supported their contentions. It is these images that tended to stand out as if testifying to the idea that “Agile” is as casual as the images appear to be.

The “Agile” meme, which can now be found in just about every aspect of modern, business, software development may in fact be quite supportive of the fluff that passes today for real software projects as is shown by its success for simple maintenance tasks. However, until we see real, critical analysis on a far, wider scale regarding the “Agile” development methodology we can assume that its realities are being ignored (or hidden) while the entire business development aspect of the profession is seduced into believing that the “magic silver bullet” for software development has finally been found.

It is a telling tale of why wisdom is wasted on the old and youth is wasted on the young…

 

 

Advertisements

About Steve Naidamast
Steve Naidamast is a senior software engineer and military historian. As a software engineer he has developed applications across a wide spectrum of platforms for over 42 years. As a military historian he first became an expert on WWI combat aviation but later moved his studies to the causes of conflict predominantly studying the WWI era.

4 Responses to “Agile Development” – The Software Industry’s New Meme

  1. tomchickory says:

    I may retire or start contracting after close to 40 years (Assembler, COBOL, .NET). Large projects and small. But dread the inevitable “how do feel about Agile, in particular Scrum” question and thinking at at that moment about the alternative endings to the “Emperor’s New Clothes” fable in which the boy is summarily executed on the spot (for the benefit of those who are not familiar, it’s a fable about a boy who yells out the truth about a totally, totally visible group think lie).

    Thanks for your post, I was searching just to cheer myself up after thinking about the Capitulation to Chaos that is called Agile.

    Like

    • @tomchickory

      Thank you for your valuable comments.

      I spent 42 years in the IT corporate environments until October 2014 when the political atmosphere in my office had become so rancid that I decided to resign and stop working in such environments altogether.

      Since then I have been trying to get a small software business off the ground. It is a hard, long slog but I am hanging in there and believe I have a good product to offer.

      The most disappointing thing when you stop working daily in the corporate environments is that what you thought you were a part of becomes rather alien to you very quickly since you are on the outside looking in more or less. Suddenly, as a senior professional, you realize just how much our profession has deteriorated since the mid 1990s. Instead of concentrating on quality application development and simplistic but efficient coding techniques you see tools, paradigms, and frameworks, most of which are either completely redundant and\or useless, bandied about as if earlier professionals had no idea what they were doing.

      A women and senior analyst of 55 years of age who was quite good at her corporate career took a new position at a company where many of the people were much younger thinking that such an atmosphere would provide her an additional understanding for all these new technologies and paradigms. Instead what she found was sheer stupidity in a microcosm. Seeing her colleagues race after these new “toys” and their project plans based on them she cautioned them to either refrain from doing such projects or refine their approaches as she had seen many times what her younger colleagues were pursuing fail time and time again.

      After so many attempts at discussing her experiences with her colleagues they basically told her to stop bothering them or her position would be in danger. In turn, like she warned, all of their projects failed for the very reasons she was trying to explain to them.

      She eventually left the company to start her own business and did so successfully.

      This is what the modern IT environments have become and their project failure rates for the most part still hover around the consistent 70% historical mark.

      I hope that you are able to find the contracting work you envision doing but between the attitudes of the younger generations and the increasing ageism in IT it could be difficult.

      Please let me know how it goes for you. I would be very interested in hearing what your experiences are.

      Like

  2. David Mack says:

    Nice article. I do not dare utter negativity about the holy agile anymore. When I was less than enthused at a recruiter gushing about a new project where “everything is going to be evangelical (yes they actually use this word) 100% agile (scrum) I seem to have since been removed from that recruiter’s candidate list. (I previously received many calls about upcoming contracts) I my early 40s I am in no mans land. Too young to retire and too cowardly to take a stab at a software business. I don’t want to be a salesman which is what you really have to be to be successful in this space. So whereas I once bounced out of bed full of enthusiasm to get to the office and create, I now dread it especially when ridiculous planning poker and other such rubbish is scheduled. The other observation I have made in the rise of this cult is many of the most avid believers are riding this fad all the way to the bank. Have an estimate at how many ‘scrummaster courses’ they are selling. This is usually a 1 or 2 day course sometimes a week which costs thousands of dollars and at the end of it you are a certified scrum ‘master’ WTF can you master in less than a week – yep you guessed it Agile. I am predicting there will be a certain saturation point and amount of time passed where there will be a gap for the next fad to come along and “outcompete” Agile leading to it’s slow decline and extinction. I wonder if I will be around that long and what it will be?

    Like

    • @David Mack

      I completely understand your dilemma.

      When ideologies take over a profession such as with “Agile” and software, the environment is the same as with the current ideologies that have infected the US political environment. You have to consistently fight to maintain your individuality and your convictions.

      It appears now that to succeed in the current software environments you must conform to the general expectations that the field is now pressuring everyone to fulfill. There are a few holdouts in this respect, which you will probably find in the large corporations considering that they change slowly and have to maintain a lot of legacy applications. However, with the younger generations entering the field, wherever they have become a majority of the workers, you will find it nearly impossible to maintain the older perspectives on software development and obtain a position.

      One excellent business manager in Silicon Valley of 60 years of age had to literally get plastic surgery to compete in his market niche. That is how outlandish the entire scene has become.

      Another woman of 55, a reputable senior development analyst, who was accepted into a company with a younger workforce saw all the mistakes they were making and tried to warn them that how they were approaching software development had already been already tried and failed. Even after their projects began failing the response from the younger teams was to encourage her to leave, which she eventually did.

      If you are finding too many doors being closed to you, you may want to consider starting a small software business where how you develop your products are completely up to you.

      Since I left my last corporate position in October of 2014 I did this. I literally couldn’t take the nonsense any more. However, this avenue is not very easy. I have two commercial products available, one since May of 2015 and I am still finding it difficult to make any sales since you have to spend so much time working the marketing aspects of your product and getting ranked well in the major search engines.

      I have corresponded with the author of probably the best .NET obfuscation tool on the market and he admitted the same issues but he is further along than I am.

      Many people have told me that it will happen if you have a good product, which I do. It just takes a good bit of time.

      I am not saying this path is easy but it is becoming increasingly the single option for those who still want to develop software without the strangulation of ideology guiding your every coding decision…

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: