Buy

Books
Click images for more details

Twitter
Support

 

Recent comments
Recent posts
Currently discussing
Links

A few sites I've stumbled across recently....

Powered by Squarespace

Discussion > Tar Pits and the mythical man month

Martin A. Thank you for that. On reflection, i'm not sure I agree entirely with the mythical man month concept. If the additional manpower is of similar merit to those unable to solve the problem I agree. But what if the new people iintroduce new expertise, expertise necessary to solve the problem? Companies do this all the time, they face a problem, say in product distribution, they realize they cannot solve in-house, so they hire this expertise in from outside. The problem, however, remains with communications, will the decision maker hear or recognize the solution to the problem.

I understood the tar pit analogy slightly differently. My take is that when a problem mires the original staff set to solve it,it attracts others (of similar capabilities) who similarly get stuck, remaking the same mistakes . However, I am arguing that if someone better equipped (say with a tractor) is attracted, they can pull the others out.

An interesting mental exercise.
Jul 25, 2016 at 11:25 AM | Unregistered CommenterAK

Alan, you yourself said

When you have absolutely no knowledge of a particular discipline (in this case software development) it is sometimes extraordinarily difficult to make sense of any part of it

Yes. What you have said there suggests that you have no a grasp of the complexity of the task involved in software creation. Think of the most complicated cooperative task you have ever been involved in and then multiply its complexity by 5000....?

Software development is not a case of 'solving a problem', where bringing in someone sufficiently bright or experienced can solve a problem that has been baffling everyone involved.

It's a case of creating something of immense complexity, by cooperative effort, where the work of one programmer (or designer) has to work correctly with the work of the other programmers. So, in principal, each programmer needs to be making joint decisions with all the other programmers, understanding in detail everything they have done, their thought processes in getting there.

Proper software engineering techniques control the complexity by partitioning the software into modules whose intercommunication can be precisely defined and then engineered independently. In cases where the software development is not managed in this way, the complexity can run completely out of control.

Your tractor analogy does not apply to software development. "The Mythical Man Month" makes the point that there is no 'silver bullet' that can be applied to save a software development project that is in trouble.

The nearest that comes to it with a software project that has become mired in chaos is to halt the project and to recommence from the start with proper software engineering discipline. However that is much easier said than done, even though it can offer the only way to avoid failure.

Jul 25, 2016 at 12:26 PM | Registered CommenterMartin A

Martin A. I wouldn't dream of discussing computer software development. I have already acknowledged my utter, utter lack of understanding. My only interest is in situations of more general interest. During my short study of this subject I was reminded of a TV programme covering the inquiry into the shuttle disaster, where the problem was identified and solved by Feynman who brought his unique gifts to play.

Now you tell me that solving software problems are different because of the degree of complexity. But the o-ring problem was also fiendishly difficult. I also don't understand why starting again is a solution. Why don't the same problems arise again or variants of the same problems appear? Don't you learn anything from failure?

To be perfectly honest I'm not exactly in the market to be re-educated as a software engineer.

Jul 25, 2016 at 12:52 PM | Unregistered CommenterAK

Now you tell me that solving software problems are different because of the degree of complexity. But the o-ring problem was also fiendishly difficult. I also don't understand why starting again is a solution. Why don't the same problems arise again or variants of the same problems appear? Don't you learn anything from failure?

To be perfectly honest I'm not exactly in the market to be re-educated as a software engineer.
Jul 25, 2016 at 12:52 PM | Unregistered CommenterAK

Alan - we are not communicating well. I have not succeeded in getting across the point that developing a sizeable piece of software is completely different in its nature from solving an intellectual problem that can be held in the mind of one person.

Why don't the same problems arise again or variants of the same problems appear?
The are certain to reappear if the same team, with the same management, is given the same task, to the same end deadline. Starting again with new management and a team, accustomed to working in a disciplined environment and using appropriate methods, with realistic timescales, can produce successful large software systems.

Don't you learn anything from failure?
No. A chaotic software organisation is not capable of learning. For a software organisation to learn, it has to be already well organised enough to avoid descending into chaos. Getting to such a level of organisation is very very difficult.

I suggest getting hold of The mythical man month and giving it a read. Its key message is that the possibility of measuring useful work (in software development) in man-months is a myth.

Jul 25, 2016 at 2:21 PM | Registered CommenterMartin A

Martin A. You beat me to it. I took the dog for a walk and thought about what you had written. I was willing to concede that complex programming could be orders of magnitude greater than I previously have had to consider and thus a "white knight" would be impossible. Before I could convey this insight, I read your last post. So you did get your point across, it was just a little belatedly absorbed by me. You must be patient with me, for me these are all new concepts.

I still do not understand why nothing can be gained from previous experience. You mentioned that really complex problems have to be solved by compartmentalization, with links between compartments established from the outset. Surely if success is achieved in some compartments, that success is not thrown away, or am I grossly misunderstanding the problem yet again?

I recall a discussion about climate models in BH where I suggested that modellers must learn from previous errors, to which you responded that the correct approach would be to start again from scratch. But are climate models that complex? I suppose an obvious reply would be that a climate model capable of doing anything worthwhile would need to be hellishly complex.

I get the feeling from you that the most complex programming is testing human abilities to organize and manage problems. Is this, in part the reason for AI development?

Jul 25, 2016 at 2:52 PM | Unregistered CommenterAK

You can't compare the software development problem with the O-ring problem without seeing that they are far different animals. Basically the O-ring thing is a puzzle. Feymann was not an expert in solid rocket boosters but he was a skilled problem solver and loved a puzzle of any kind. He had to find out how it worked while he was solving the physical problem and more importantly the problems with NASA management which had allowed the disaster to happen.

Software development requires skill, management and organisation of a different kind. And routine grinding painstaking work. I've shared an open-plan office with a software team and seen how it happens, but when they invited me to learn the language (PL/1) and join in I declined. I'm a puzzle solver too. I like a problem which will be finished when I find the cause. I'm good at it. Find out what is happening, how it's supposed to work, find the problem anf give back to the programmer to fix. Sometimes in a language I didn't know (this was back in the day when you could read code). If like me that is your skill set, you can be a fixer but not a creator.

Our lead programmer used to go down the pub at lunchtime and sleep through the afternoon. Nobody minded because of his productivity in the morning. Another person I've worked with caused a colleague to comment ' When Charlie's in it's like having three men off sick.'

Jul 25, 2016 at 3:04 PM | Unregistered Commenterrhoda

rhoda. You may well be correct - in fact I'm sure you are. I also see myself as a puzzler (I don't always solve my puzzles) and perhaps also as an ideas generator. The last usually because I'm essentially lazy and want to take short cuts. My last few research papers were on puzzles I identified and failed to solve. Most papers in my field believe they have solved the task they set out to do. So I submitted my efforts very hesitantly (one took me more than ten years to send off) and was surprised and gratified at the receptions they gained.

I am not that interested in computer programming as such, but have been intrigued by the application of these (to me) new concepts to complex problems but of a lesser degree of complexity. Thus the tar pit analogy does I think apply also to very difficult problems where the original team set to solve them are floundering in the pit. Adding more of the same manpower won't solve these problems but the addition of a "tractor driver" might.

I think Feynmans brilliance was also due to his abilities to talk to, and listen to others of varied background. After all he listened to a female astronaut as well as engineers who were convinced the o ring problem was not due to excess vibration. He also possesses great communication skills. He belongs to the group my better half and I call obituary people - people with so many positives that you don't know all of them until you read their obituaries.

Jul 25, 2016 at 4:07 PM | Unregistered CommenterAK

I still do not understand why nothing can be gained from previous experience. You mentioned that really complex problems have to be solved by compartmentalization...

Alan - I don't think I did say that. I think I said that that large software systems (I think I did not say 'complex problems') have to be partitioned into manageable modules.

with links between compartments established from the outset. Surely if success is achieved in some compartments, that success is not thrown away, or am I grossly misunderstanding the problem yet again?

That's true. But in that case you don't have a system whose development is out of control. Just some modules that need to be redone and which the project manager will ensure get done.

I recall a discussion about climate models in BH where I suggested that modellers must learn from previous errors, to which you responded that the correct approach would be to start again from scratch. But are climate models that complex? I suppose an obvious reply would be that a climate model capable of doing anything worthwhile would need to be hellishly complex.

I am delighted that you recall my saying that.

It's not so much the complexity that means that climate modelling needs to be redone from scratch, it's the quality issue.

If you have software full of errors (which in itself is a sign of chaotic development) it's pretty futile to try to eliminate them in the vain hope of obtaining a quality product. It's a good guideline that the more errors you find and correct, the more errors you can be sure still remain in the product.

In an analogous way, modelling something complex like climate depends on the correctness of what has already been done. Starting from scratch, with the level of quality control - validation - of models that you would require in a system upon which life depended, then you could have knowledge of the validity of the models. If you build on layers that might be just about ok or could well not be, then what you finish with is not worth having - other than as a curiosity.

I get the feeling from you that the most complex programming is testing human abilities to organize and manage problems. Is this, in part the reason for AI development?

AI was a big thing in the 70's and 80's but was by and large essentially a giant wank, with extravagant promises for what it was going to make possible. I think it was very generously supported by the EU.

Jul 25, 2016 at 4:15 PM | Registered CommenterMartin A

Its good to see this discussion on program development. Over the decades many methodologies have been dreamed up to ease complex program developments.

Anything that you dream up can be implemented in a program (a big statement I know) on the basis that you know how the system you want, works.

Back in never never land, our favourite climate modellers, do not yet fully understand how the climate works, how each variable interacts with each other variable, and even do we know what all of the variables are.

So in the climate world, currently, it matters not how their complex programs are designed and implemented, and will not, until we know all of the underlying processes.

Jul 25, 2016 at 5:34 PM | Unregistered CommenterSteve Richards

"AI was a big thing in the 70's and 80's but was by and large essentially a giant wank"

Artificial Intelligence is not a cure for Human Stupidity - despite what the wankerati may tell you.

Jul 25, 2016 at 6:06 PM | Unregistered CommenterRL

I am a software developer and I work as a contractor with my own ltd co. Almost every contract I get is to fill a knowledge gap or meet a looming deadline.
The solutions to a problem are extremely variable, there is always more than one way to skin a cat. Quite often the tune might be the same, but the words are different.

Jul 27, 2016 at 2:44 PM | Unregistered CommenterEternalOptimist

EO
In the days when I worked on a variety of developments throwing people at a late running project late in the day was guaranteed to delay the work further. Each additional person took one of the team off development and into training. This was true no matter what level of skill and experience was being drafted in. As Martin A says about other issues the late recognition of problems is probably a symptom of poor management.

A poorly performing large and complex system is beyond saving and needs a complete redevelopment from requirements specification upwards. Climate models fall into this category.

Jul 27, 2016 at 3:59 PM | Unregistered CommenterSandyS

Aye Sandy.
I would never pick up a half finished job, I would just plain refuse. My standard line is 'I can finish that off in nine months or I can start from scratch and do it in three'

The problem with crummy half finished projects that the 'Owner' wants you to finish for them is that they are basically asking you to implement their solution for them. The three month option is me saying that I will implement my own solution.

I don't see the failure of climate models as an IT implementation problem. They would have still been heaps o shoite if they had been worked out with a pencil and paper

Jul 27, 2016 at 5:20 PM | Unregistered CommenterEternalOptimist

@Martin A, Jul 25, 2016 at 4:15 PM

AI was a big thing in the 70's and 80's but was by and large essentially a giant wank, with extravagant promises for what it was going to make possible. I think it was very generously supported by the EU.

Love that description which sums up my feeling about doing it at Uni in mid '80s.

AI is still being claimed to be the next big thing in x years - rather like fusion.


AK,

Software programming/engineering is very different from most tasks. It requires adherence to rules & requirements, but also creativity and lateral thinking.

One section of the population that is over represented in programming is those with varying degrees of autism.

BBC Four's The Joy Of Logic explored how we think.

It revealed I am a Vulcan like Spock.

Jul 27, 2016 at 9:04 PM | Registered CommenterPcar