Software Engineering: The Sociology of Software Project Failure

Recently a new report has come out stating that recent trends in software development have shown a rate of failure at approximately 68%.  Such failure is not judged simply by the fact that a project failed to be implemented go in but instead among several factors, which include unsatisfactory results, missed implementation date, budget overruns as well as project cancellation.  Given this criteria, it easy to understand why the percentage of failure is so high and why it has remained hovering at this rate since the inception of modern software development.  In other words, it is a stark indication that the Information Technology field has yet to mature like every other technical or process-driven field (ie: manufacturing).

When reading such reports the analysis of project failure has consistently revolved around what development processes were used or not used and the number of software engineering mistakes that were made; all of which by now have been substantially proven by SE analysts over many years of studying project failure as the cardinal mistakes technical managers constantly and consistently make.

In fact, the reporting style of such an indicator in the Information Field is so formula driven that one report hardly reads as any different from another.  Its as if such reports are simply cranked out on a random basis with the same reasoning and the same failure numbers just to keep reminding professionals in the industry that it still stinks when it comes to quality output.  The interesting thing to note is why such reports are never changing and have been consistent for some 35+ years in a field that promotes modernization of technologies?

Software development is not done in a vacuum; it is part of a larger process that must include several parties coming together to produce a deliverable; the users who will eventually be using the software, the technical managers who will be providing guidance to the projects, and the developers themselves who will create the actual product.  With three distinct parties all who have vested interests in a successful outcome, you would think they would do everything possible to make sure their projects are successful.  However, this is hardly the case.

The “ugly truth” is that actual software development is done within the constraints of a corporate culture that supersedes the interests of the parties to any individual project.  And it is this culture that provides an underlying sociology that most often causes software project failure even before any project is actually begun.

As a company becomes increasingly larger its internal bureaucracy naturally grows along with it, which in of itself becomes a self-perpetuating entity, that most often invades the corporate landscape with its own agendas from an institutional perspective. This group-like agenda has as its underlying foundation the agendas of the company managers from the most senior down to the middle layers are most often more interested in their own self-aggrandizement than the best interests of the company.  The management hyperbole about only promoting the finest quality with in any specific corporation is complete nonsense as such statements are completely disassociated from reality.

To understand such a contention one also has to have an understanding of the sociology of corporations, which few people do with the exception of those that study such institutions.  There is a difference between a company and what we have come to know as the “corporation”.  Companies, especially private ones have been around since the 17^th century and earlier.  They historically have been made up by people who have a vested interest in either providing a quality service or a quality product.  And if they failed there were no shareholders to worry about though the owners of course lost their shirts in the process.  “The Company” as we should call it is the actual basis for modern Capitalism.  They were often small to middling in size and provided societies with cottage industries that were often competitive within their own right.  Some were large such as the famous Lloyds of London but they prospered since their managers were sincerely interested in maintaining a sense of quality about them, which is what also provided them with their reputations.

“The Corporation” on the other hand, the type of institution that we constantly read about on a daily basis was actually created as a scam, a swindle in the 17^th century (see Joel Balkan’s excellent sociological study on this, “The Corporation”) with the sole purpose of fleecing third-party investors out of their monies.  So blatantly flagrant were these situations that European governments banned such institutions from operating for over 200 years.  It was in England where the ban was eventually first lifted but only to the extent that “corporations” could not extend their risk to unknowing shareholders and all owners in a corporation (including the shareholders) incurred legal liabilities for any illegalities committed by the corporation.  This then kept such operations in line as far as fair and equitable operating practices were concerned.  And up through the late 19^th century both companies, which were still private, and corporations which sold shares of stock to shareholders for capital investment operated side-by-side without too many upheavals, if any at all.  Then two things happened along the way to the modern misanthrope we now understand as the “corporation”.  In and around 1898, the corporation was granted “personhood” when a judge granted such a consideration to a corporation involved in a litigation a judgment that was based on extraneous trial documentation, which was not supposed to be considered allowable for interpretation due to the nature of its position in the document it was found in.  The judge in the case thus allowed a corporation rights as an individual, which set precedent for all other such litigation up through the present.

Recent studies of the case have also concluded that the allowance of “personhood” was inappropriate and not actually legal thus in essence invalidating every such instance where a corporation was treated as an individual in the case.  However, try undoing over 100 years of jurisprudence…
The next thing to happen was around 1925 when corporations were able to eliminate shareholder liability in terms of illegalities committed by the corporation.  These two events allowed the corporation to separate itself completely from the people who were actually running it thus eliminating any personal liability for anyone involved.  It was a boon for the institution as it was now possible to return the “corporation” to its roots as nothing than what it was originally designed for, fleecing people out of their monies.  With the known reasons for the current economic downturn there is not really any possibility of disputing this.

From an institutional culture in terms of the private company that operated within a Capitalistic structure that was more or less fair and equitable as a result of the desire to produce a quality product we went to the modern corporation whose sole motive is the creation of a profit.  Eliminating that sense of quality has had terrible effect on what corporations thus produce and the evidence for this abounds on a daily basis.

To believe that software development, which came on the scene long after corporations were in a state of sociological decline, could produce quality output is an almost impossibility.  And here the evidence is quite extensive as well.

This is not to say that there aren’t companies that still produce quality output and that there aren’t companies that can produce quality software.  In both cases there are.  However, such cases are fostered by only two factors; either the company must produce quality products by necessity or management is enlightened enough to understand that quality output will always outlast those competitors who do not provide such output.

The classic example of necessity is the aircraft manufacturer.  If Boeing Corporation for instance produced planes that crashed, even on occasion, they wouldn’t be in business.  And it is interesting to note that the Boeing 757 (and its big brother the 767, which was modeled off the 757) is considered one of the finest commercial manufacturing\software projects in history producing an aircraft of uncanny performance with an exceptional level of quality that is very difficult to equal or match.

However, such quality is not often found in the environments that most software developers and engineers work in because most corporate cultures don’t really consider it important.  The result is the newly described term for acceptable software development in most companies, “good enough”.  This result is merely an extension of the l level of quality that most sociologists of corporate institutions have attributed corporations of being capable of, which is simply that of “mediocrity”.

It is no joke when people state that the companies they work in have nothing better to offer than mediocre products.  Often they are quite right and simply coming to an understanding what most modern companies are truly capable of.

So this is what the underlying basis is for where all software project failure begins.  However, in terms of the actual corporate Information Technology organizations themselves, they have been literally ravaged by uncanny business theories and implementations that have no regard for the requirements of software development but are simply implemented from management whim or the corporation’s need to “cut costs” in order to look good for Wall Street.

Information technology costs… period!  There is no getting around it but corporations in their own enlightened manner continue to find ways to reduce these costs, which drive down the numbers on the ledgers, while driving up the “hidden costs”, those costs that cannot be accounted for unless a sociological study of the corporation has been implemented.

Outsourcing is one of the most outlandish processes to have ever been implemented by a corporation.  The toll in dislocation that tends to spread across the entire spectrum of a company as result has been massive destroying vital corporate loyalties while reducing quality to such an extent as to make it a wonder that such companies survive at all.  Again the evidence of this is quite extensive and needs no further explanation.

From a military analytical perspective, (I am a military historian as well as a senior software engineer), the concept of outsourcing goes against every military axiom of logistics ever founded making the term “business leadership” a classic oxymoron.  Outsourcing unnecessarily extends a supply-line and the longer a supply-line the more vulnerable it is too disruption.  How many times then have developers complained of having to deal with people in another country who can barely converse properly over project specifications?  That in of itself is a major disruption.

Yet, outsourcing is done to contain costs, or so the mantra goes.  It is not done to produce a higher quality out- come since when you reduce costs, at some level quality must suffer.  And for any software project there must be a minimal level of cost in order to produce quality.  However, when managers are making such decisions they often respond to such needs by promoting the hard work of the software development team.  To them such “hard work” will make up for any of the deficiencies they have provided the development environment with.

Unsurprisingly, most American managers are bone stupid.  Not all, but most.  This easily filters into the Information Technology organization within the technical management cadres that most often haven’t a clue about what they are managing.  American IT technical managers, though they used to enjoy a good reputation have never really deserved it.  Most often they have been nothing more than abstractions of the general business executive that often rules from a tyrannical vantage point.  The really good technical managers in the field have been often ground under as they attempted to bring a sense of reality to project development which often went contrary to the predicted deadline for project completion.  It should be remembered here that we are not talking about companies where quality is a necessity or the management is enlightened enough to require it…

This has been a prevalent problem in American Information Technology and it has become worse with the influx of Indian technical management into the management ranks of US corporations.  Through the 1990s I always found my Indian colleagues to be some of the most professional technicians I have ever had the pleasure of working with.  However, after the dot-com bust in 2000 this seemed to change as “in-sourcing” less expensive professionals from India and other third-world nations was ramped up significantly.  From highly professional and capable cadre of development specialists from India suddenly the people coming over were of a different level of quality, while their own management contemporaries were being increasingly hired as managers in US firms.  This brought two sets of new problems to the corporate Information Technology organization; one of which has been often kept hidden from view and the other which is impossible to hide.

Given the spiritual nature of the Indian people it is very surprising to find that in terms of their business and politics it is one of the most corrupt nations on Earth.  Yet, this aspect of Indian society has been well documented internationally and by their  own investigative journalists.  Its corruption suffocates legitimate business endeavors to the point that it is a wonder that any such endeavor is even successful.  In a broad review of several major business centers in the world several years it was surprisingly found that India ranked the lowest in terms of business-friendly environments.

What is heard about Indian prominence in terms of their business acumen is in reality only a handful of companies when you compare it to the entire population.  And in terms of Information Technology and its associated adjuncts such as technical support, the working conditions for the people there are equivalent to what was once an abomination in US labor up through the 1960s.  Not everywhere of course but enough so that mainstream business news has picked it up often enough.  Indian call centers have a reputation for being nothing more than ‘sweat shops” while small farming in India is so ill supported that many simply commit suicide out of despair.

What such environments have yielded are business managers from the more affluent classes that are often arrogant and completely devoid of any concern for the people that work for them.  And these are the people who have been hired into many US technical management positions.

Indian technicians on the other hand have seen a reduction in quality from those who were coming over prior to 2000.  With the chance to increase their outsourcing companies, Indian technical training, which was generally superb began substituting the equivalent of less than quality training for people being sent over to the States to take positions as consultants from these companies.  Reports abounded of such personnel being trained just well enough to pass interviews.

Though I believe some of this is exaggeration, Indian technicians today do not seem to have the same grasp or understanding of what it takes to actually develop quality systems as their predecessors did.  And their coding seems to demonstrate this whereby sloppiness appears to be a more common trait.  This is not by any means an indictment on Indian technical capabilities.  They are very capable personnel.  However, their training, which appeared to be more robust prior to 2000 now appears to be centered on memorizing technical facts and attempting meld strictly factual information with a development process that requires a sense of flow.  System and application development is much like a dance and contemporary Indian technicians appear to be so immersed with the clinical aspect of such development they have seemingly learned to ignore the movements…

This is not strictly the fault of the Indian technology industry by the way.  In some respects, Indian service companies are merely responding to what American corporations are seemingly demanding… cheap labor.  And the cheaper the labor is, the lower the quality output you are going to get.  It doesn’t matter how you slice and dice it from the perspective of a given labor market.

These same problems have plagued Asian outsourcing as well but for some reason, few Asians are hired into American technical management positions as much as their Indian contemporaries.  This may be a result of the more predominant Indian outsourcing firms whereby Asian developers are seen at an even lower rung in the “food chain” than that of Indians.

However, surprisingly this does not seem to be symptomatic of Russian developers or their firms as Russian technicians demand higher pay while also expecting to be treated as equals in the development process whereby they can recommend and implement their own proposed methodologies for accomplishing a given project.  The result is that Russian outsourcing is no way comparable in breadth to Indian or Asian outsourcing.

All this was demonstrated in a study of the three major outsourcing regions in an extended article in Business Week several years ago.  There is no evidence so far that this situation has changed to any major degree, especially with current economic circumstances.

What is also interesting to note is another common phenomena within American corporations (and no doubt the UK ones as well since they have been following similar trajectories) that has been occurring since the Reagan Administration; the rise of psychopaths in the management ranks.  In their critical treatise on the subject, “Snakes in Suits – When Psychopaths Go To Work”, Drs. Paul Babiak and Robert Hare, the two psychiatric researchers that created the definitive screening process for weeding out such people within institutions, demonstrate that increasing numbers of classic psychopath personalities have been appearing in US management ranks since Reagan took office in 1980.

This can be attributed to the constant reduction in regulatory requirements that corporations no longer must adhere to along with the rise in prominence of the business manager when compared to technical specialists, scientists, and doctors who have all seen their professions downgraded to subservience to business interests.  This combination of events have encouraged Human Resource departments to more often ignore due diligence in their screening of prospective employees thereby allowing power hungry personalities to enter their management ranks.  And it is only later that the level of damage incurred by these personalities has been found out where in some cases the level of damage has reached such levels as to literally destroy companies and their reputations.

This is not a joke and is increasingly being unknowingly found out by staff members who are finding themselves at odds with their own reactions to their superiors when they feel like they have just experienced a classic “nut job”.  As both Babiak and Hare would say, “They most likely have…”.

This new wrinkle in the corporate management fabric of many US corporations has become such a serious problem that it warranted the development of a screening process as both Babiak and Hare have developed.

Much more can be discussed regarding the detrimental idiosyncrasies of American management but suffice it to say that much of this management has serious personality disorders which often lead to the precarious decisions that have been made under the aegis of “cost containment”.

All of this and more has a massive ripple effect into software development by setting unrealistic and ruthless attitudes that technical management takes towards getting software development accomplished without realizing or even caring about the requirements of the craft…  And it is a craft and much more so than a science in many respects.  However, the additional stresses that are placed upon developers and engineers aside from the management stresses they labor under can be directly attributed to the software vendors themselves.

Again, such companies are driven by profit and set their priorities accordingly by rushing out  increasingly complex sets of development tools with new defects that all have their own learning curves.  For  an industry that one would think would be run from a somewhat of a scientific perspective, instead vendors produce new software at a rate that has long exceeded the capacity for technical professionals to absorb and work with efficiently.  The effect is to reduce the overall knowledge-base that is vital to good software development.  Thus software developers and engineers labor with tools that are in a constant state of flux where differing versions of even the individual elements have defects that must be surmounted.

Microsoft, one of the most egregious causes to such a state of chaos in the industry releases new versions of their popular .NET development environment every few years which causes a complete imbalance in the professional technical arena.  Corporate developers slowly integrate such new upgrades putting such professionals at a disadvantage when compared to their contemporaries in the consulting and freelancing arena who most often look to use the latest technologies available… and with good reason.  The fickle employment market in Information Technology often demands the latest skills while rarely if ever taking real advantage of them.  Nonetheless, the misappropriated confusion that comes out of corporate hiring specifications of demonstrates the complete disconnect between what is really needed in a good technical professional and what more unrealistic perceptions often produce in terms of specific hiring requirements.
A classic example of this serious disconnect was the competition in the 1990s between technical personnel who were still developing with Microsoft’s Visual Basic version 5.0 and those that were working with the newer 6.0 version.  With the exception of a few features, both versions were basically the same in capability and yet, companies were reluctant to hire people who were still primarily working with the 5.0 version.  It was a level of hiring insanity that to this date has not been equaled.

The ever increasing production of software upgrades and new versions with ever more extensive features produces a level of stress on software engineers and developers that has few if any rivals in any other industry.  If you don’t have access to the new technologies in your current position you are forced to spend additional hours studying in order to keep pace with the market.

All of the factors discussed provide a level of influence within the Information Technology organization whereby frustration abounds along that combined with ill-equipped and ignorant management as to the proper practices and techniques for good software engineering has maintained the IT industry at a level of immaturity that will continue to produce high rates of software project failure while at the same costing business hundreds of millions of dollars in lost productivity and wasted efforts.  And this is all of a result that business itself will not reflect on what it is doing to itself or the people involved.

American business over the years has become the worst example of poorly run organizations that attempt to align themselves with military style theory without actually understanding what such theory involves.  A study of the many different doctrines of business organization a number of years ago found that out of all of them, American business organizations have overwhelmingly adopted the “positioning doctrine”, which based off of Sun Tzu’s famous treatise on warfare, “The Art of War”, attempts to provide senior management a set of protocols that only investigates and considers decisions based upon factual data.  Unfortunately, if this were so, many of the serious issues that plague US corporations and to a lesser extent their international contemporaries, would not be as disruptive as they already are.

With this type of upheaval a continuous factor in the daily endeavors of IT organizations it is near nigh impossible for good technical management to implement policies and practices that have been thoroughly proven to yield cost effective and successful project implementation while substantially reducing the costs to their companies.  However, with the limited and narrow focus that the majority of business managers take they are incapable of understanding the realities that so many business organizations are in desperate need of.  The current financial crisis is merely a stark clarification of how poorly most US businesses are currently being run and thus the hope or even the expectation that quality software engineering will become common-place in such environments is nothing more than a pipe dream.

Thus, software project failure will continue to remain at the 70% average as it has since it was being first analyzed and calculated…

Black Falcon