At Priocept we believe that an engineering approach, and a deep understanding of software engineering in particular, are critical to the success of any digital product development, e-commerce, content management or mobile project.  In fact, software engineering is critical to any project whose success depends on the application of software technologies to deliver a new product or service.

So in this article, we go back to basics to try to explain; what exactly is software engineering?  And we look at why is there so much software development going on in the world that does not apply a proper engineering philosophy.

Engineering vs Science

Firstly, for background, we should look at the difference between Engineering and Science, as they apply to all fields and not just software.

Science is the study and development of knowledge of how the world behaves, according to the laws of nature.

Engineering, on the other hand, is the application of knowledge gained through science, to make things happen that would not otherwise occur in nature.

This distinction is quite clear if we compare, for example, Physics and Mechanical Engineering.  Physicists concern themselves with the study of how the world works, and how the forces of nature behave.  Whereas Mechanical Engineers are continually endeavouring to build contraptions to do clever stuff that would never happen by accident in nature.

The Physicist might work out laws of nature relating to fluids and mechanics (Bernoulli’s Equation, Newton’s laws, etc.) but it is the Engineer that will take this knowledge and use it to build an airliner or a skyscraper.

Computer Science vs Software Engineering

In the digital world however, the world of computer hardware and computer software, the distinction between science and engineering is not so straightforward.

We have Computer Scientists and we have Software Engineers.  Because computing is a new field (less than 100 years old compared to 1000’s of years for science and engineering in general), the distinction between Computer Science and Software Engineering is not so clearly defined or even commonly agreed.  But we can say that Computer Scientists are busy studying the domain of computing to further knowledge in this field, while Software Engineers are busy building software systems that do useful stuff for people in the real world.

We probably can’t say, however, that Computer Scientists are studying nature, in the way that Physicists or Chemists are, because computing concepts (for example data structures or network protocols) don’t really exist in the real world, they only exist in an abstract way.  Arguably, this makes Computer Science more like a branch of Mathematics than a Science.

Software Engineers vs Developers and Programmers

So we have Computer Scientists and Software Engineers.  But then there are also “software developers”, who used to be called “computer programmers”.

In fact, we probably have millions of people across the globe developing software, who are “software developers” but are generally not “software engineers”.

So What Makes a Software Engineer?

If you are building an airliner or a skyscraper, then you are going to have to follow a very well defined and very well structured process to achieve your end goal, or things are going to go badly wrong very quickly.

In the most general terms an engineering process looks something like this:

1. Research and Conceptualisation – gathering information, facts and data, and defining the context, concepts and vocabulary that will support the engineering steps that follow.
2. Requirements and Feasibility – assessment of what needs to be built, and whether it is possible and/or viable.
3. Preliminary Design – high level design including definition of an overall architecture that breaks down a complex system into separate but interdependent sub-systems and interfaces between those sub-systems.
4. Detail(ed) Design – elaboration of the high level design into a full description of the system such that it can then be built in the real world.
5. Prototyping – building the first version of the design in the real, physical world.
6. Manufacturing (Production) – building the final version of system, as per the design and prototype, often in large quantity.  This includes building (engineering) yet more additional systems (manufacturing tools) that you need to manufacture the system itself.

So we can say that anyone that follows a structured process that looks anything like the above, is following an engineering process and is therefore an engineer.  And if this process is being following to produce a software system, then that is software engineering process.

The Lack of Engineering in the World of Software

But the engineering discipline often starts to fall apart in the real world of building software.  Why is this?

Firstly, the distinction between design and production is arbitrary, perhaps even non-existent, in the world of software.  Why?  Well, manufacturing is the duplication of a prototype into one or more (sometimes millions of) production units.  And in the world of software, the “manufacturing” step is trivial (let’s ignore software deployments and server infrastructure for the moment).  You used to have to burn a CD, now you just copy your software over a network, at zero cost per unit.  So writing the software is effectively also the manufacturing.  But it’s also the design.  Already we have merged two key, distinct, and traditional engineering concepts into one, and lost the opportunity to control them independently.

Secondly, the risk and cost of design faults in production is often low in the world of software.  If you build a bridge and it fails, the impact is very high.  If you manufacture ten thousand cars that have to be recalled due to a design fault, the impact is very high.  This leaves little choice but to follow a very structured engineering process to reduce risk and ensure quality, before production starts.  But in the world of software, you can often just get away with fixing the thing after it is out there in the wild – upload some fixes to the server, or send your customer some patches.  This encourages the bypassing of important engineering processes.  Of course, if you are writing the fly-by-wire software for the Airbus A380 or Lockheed F35 then you will probably want to stick to the engineering lead process.  But for your average website or software application, it is tempting and all too easy to avoid key engineering steps and deal with the consequences later.

Thirdly, the supply vs demand equation for software engineering skills is badly out of balance.  There is a huge global growth in the development of software systems, which is way out of kilter with the rate at which well educated, skilled and experienced software engineers can be trained and developed.  This problem is made worse by the computing and software industry’s failure to establish credible professional qualifications that compare, for example with Chartered Accountant or Chartered Engineer status.

Fourthly, there is poor understanding and little appreciation of the craft skill of software development.  A talented Mechnical Engineer may design a sophisticated and innovative machine such as supercar, but he is utterly dependant on a range of craft skills to finalise the design and put it into production.  These skills include metal fabrication, machining (turning and milling), welding, composite fabrication, plumbing and hydraulic fitting, and so on.  Without high levels of craft or “hands on” skills, we will have a beautiful design on paper (or CAD), but will end up with a shoddy end product – components that don’t fit properly, welds that are weak or aesthetically poor, composite structures that delaminate, and so on.  In the world of software, we will have a beautiful design in UML, wireframes, or other Visio masterpiece, but again a shoddy end product with inconsistencies in the interface layout, lack of robustness or protection of the user from error conditions, confusing or unresponsive functionality, and so on.  In our opinion, high levels of craft skills relating to building of software products (particular the user interface aspects), are second only in importance to the software engineering process itself.

These four factors combine to put us in a world where probably 50% or more of all the software that is produced today is not “engineered” at all, and is often not “crafted” either, but is simply “boshed out”.

For all the industry talk of “requirements analysis”, “use cases”, “enterprise architectures”, “agile development”,  “user stories” and “business analysts”, the fact remains that millions of lines of software code (probably billions, to be honest we would rather not think about it) are produced every year without a single bit of engineering (or craft) to be found in them.

This results in:

1. Software that is unreliable.
2. Software that is difficult, unpleasant, and sometimes impossible to use.
3. Software that is extremely difficult to evolve or improve.
4. Software that is hugely expensive to run or maintain.

The only solution to this problem is the application of proper engineering discipline and highly skilled software “craftsmen”.

But it is Still Early Days…

Hopefully this article has helped to illustrate why engineering has been so pivotal in creating the world that we live in today, from bridges to airliners to internal combustion engines, and why the world of software development has a long way to go to catch up with other engineering disciplines that have already had time to evolve over hundreds of years.

The sooner we move to a world where businesses treat all their software development projects as engineering exercises, which means they must be managed and executed by engineers (just like Boeing or Toyota’s development projects are run by project managers who are invariably trained engineers), the sooner we will move to a world of consistently high quality software, and consistently high quality experiences for the end user.

In the second part of this article we will investigate some of the world’s most successful software and digital technology companies who, without exception, apply solid engineering principles throughout their development projects.