I was a DoD contractor (the coding type, not the shooting type) for almost 15 years, and this is a really interesting read. I've heard some software devs state that "the waterfall method was never really used". Those devs were not DoD software contractors for almost 15 years. DoD was and is all about the waterfall model. I have spreadsheet PTSD from seeing so many waterfall diagrams and project manager's Gantt charts. I was working on a peanut-size project for the Navy (around $5-10million over 5 years) when Agile began picking up steam (mid 00s). I introduced the team to it and we began doing Agile to decent effect (at least in comparison to waterfall), and introduced it to our Navy sponsors. I remember some higher-ranking individual ask, "wait, so this method can actually deliver software?" Or something like that. Of course, Agile is no panacea, but the iterative nature was waaaay better than what the DoD was doing.
I really like how they talk about "individuals of talent" in the document. Much of the DoD focuses around process, to great effect. The problem is that you can't take a process and put software drones on it and deliver software of any complexity. That goes against much (but not all!) of the DoD paradigm, it is interesting to see they might be changing.
Of course, they will meet great resistance, after all, the first job of a successful contractor is to extract as much money as possible from the government with as little work as possible. "Never finish a contract" was the mantra we had learned from the giant contractors.
Glad to be out of there, but it was a really interesting experience!
I keep banging on about "software is a new form of literacy".
Imagine you have a group of illiterate managers who have just hired a group of writers to write some company policy documents.
You could have the managers (carve / draw / modern dance) their requirements in a waterfall manner.
Of course you could get one of the managers to sit down with the developers and tell them how the policy's work each day, embed with them Agile style.
However in the end you can't be everywhere, sometimes there aren't enough people in the company who understand what is needed, or even be sure that the embedded people agree on what is needed.
These are business problems - but they are also literacy problems. We have a literate society and so there are enough literate people to write policy documents to cover a whole compmay.
We need to make enough people who can code, so they can write the policy documents of the future.
This is an interesting take. I've always rolled my eyes at the people who say "everyone needs to know programming" but when you put it like this, yeah, why not?
The only thing I'm not convinced about is whether programming is something of a functional specialisation or if it will (GAS already?) become a literacy-like thing.
I mean, we generally think of numeracy as something widespread in the West for the past few decades, but the average person really does not do even basic arithmetic very well. Are we really going to do a better job of teaching people programming, or will we have to resign to the reality that most people won't learn it well?
My take on this is the basics of astronomy. The history of how astronomy slowly worked out the earth goes round the sun, distances etc is fascinating (#)
The bit that's gets me is how fast does earth spin (without which sun rise is a bit hard to explain). Until it was accepted that air was just a thin atmospheric layer, it becomes insane to imagine we spin at a 1,000 miles an hour though a air medium that extends to Jupiter. How do you keep your newspaper still.
Kids today have Mickey Mouse cartoons explaining space rockets, moon to mars, earth spinning etc. We just accept it as true without trying to be clever about it
The same is true for literacy and numeracy - just having a society where you absorb character arcs, sub clauses, chapters, publishing, versions etc makes so much difference. Mathematical symbols were the names of Ed Sheeran albums.
Look at something simple like version control - the idea is fairly simple and hot is almost perfect for it, but oh god have you tried to use Sharepoint. Once you are in the WYSIWYG mind set, version control breaks horribly and most most business users should have this all the time but it's not in their world view. Explain it to them and they say "but how does your nreasoaper not flap around in thousand mile hour winds"
(#) Look for episode on the podcast "Subject to chnage"
I've been harping on this for decades. Back in the early 2000s there was STILL business news interviews with some CEO bragging about how he has his secretary print out the emails and doesn't know how to use a computer. Thankfully that ended probably around 2005.
Fully 25 years after computers-in-business, especially for any significant size of business, was virtually universal.
Humans don't manage businesses anymore. They are essentially all software managed. Business management is now really meta-management of business management software.
I haven't read the doc, but I worked on lawsuits (I'm a lawyer) that at their core were asserting that agile software development isn't as safe or bug proof as waterfall. Basically in the medical device (or adjacent) space the idea asserted was that 'fail fast' was illegal and didn't have enough safety checks.
Agile software development is a tough pill to swallow for non-engineers. The lawyers and business people can't wrap their head around the idea that building iteratively could actually reduce bugs and improve quality - because the way they work is via process planning docs and checklists.
I think one way to go about it is to just put the Agile bits inside a Waterfall.
Which is frankly how it works anyhow.
You can characterize early sprints at 'research'.
Later the sprints are just 'sub projects'.
I think people could be convinced that 'feature refinement' is a real thing. Put it out in the field, doesn't work the way we want it, and so it needs to be refined.
I know that I would hide 'Agile' to anyone outside the team and just talk in simple, easier to understand phases.
It's also possible to put checklists in those phases, and if you say someting like '75% features' at such and such phase and assume '25% of that 75% will be reworked' it's not irrational either.
Building iteratively can reduce bugs and improve quality other things being equal. The problem with how Agile methods sometimes get implemented in practice is that an excess focus on quick-and-dirty iteration to the exclusion of everything else leads to having mountains of very poor quality code, with little to no support structure around it in the form of broader design or documentation. Agile methods advocate for growing these kinds of support structures via refactoring, but this is often not done to anywhere near an adequate extent.
I think this is true even in “civilian” industries. Can say doing this for over a decade now that I have worked at exactly one place that followed anything close to what was laid out in the Agile manifesto.
Every other place always sings the praises of agile and/or scrum but it’s always just waterfall with ceremonies.
I feel like the collective of how software is developed evolved little from circa 2006
The notion that gets talked about at Netflix is “context not control” and I think it would go along way if everyone looked at software development management this way. Too many people try to control software development lifecycles instead of just providing engineers enough context to get their work done and trusting that reasonable people will make reasonable decisions
IMO it’s not this simple. Culture has more to do with it than anything. See my note about Netflix having this mantra of “context not control”. Managers are less inclined to be domineering in that setting and allow people to do the work in a reasonable way because they are given the context of why their work is being done in the first place.
Not perfect but a more than above average way to approach the problem yet this kind of thinking or its permutations have not caught on at all. It’s still top down process oriented models
Agile is a tough pill to swallow for many engineers, but it's become part of the theology of software development, so everyone's afraid to object to it.
This is so true. They don’t teach agile in management classes and ‘planning’ generally means years, not t-shirt sizes. I believe that also many economic theories and practices don’t regard agile practices. I’ve always felt that the business and their planning are at odds with the agile ways.
It would seem they don’t do a very good job teaching how to think about uncertainty, risk, and unknowns, which is what agile is good at addressing. If you want a suit to lose their mind, tell them they can only pick two of feature, quality, or schedule.
Waterfall seems a lot better when you have done something before and there's nothing to discover.
> Waterfall seems a lot better when you have done something before and there's nothing to discover.
I've said this here on HN before, but the only Waterfall projects I've seen it work reliably were where:
1. The systems were small (in which case any method would work)
2. The systems were trivial (in which case any method would work)
3. The systems were well-understood by the people making them (in which case any method would work)
It does not handle large, non-trivial, or novel systems very well. Almost without fail they will deliver late, over budget, or deliver the wrong thing. This can be a win for contractors, though, who get to bid on the rework or sustainment contract for their own work.
In my experience it was less a lack of intelligence and more results-oriented thinking (they thought we were wrong so they found a way to argue it).
If you have twenty versions of a thing and multiple deployments to fix and fine tune it, that means to this person there are 20 plus examples of "mistakes" you made in building it. They were looking for evidence that the software was built with "poor quality controls" and ironically interpreted iteration as evidence of poor build quality.
I’m betting releases and iteration cycles they’re tied to. The most agile place I ever worked shipped code most days of the week in some fashion. For control reasons we released Tuesday-Thursday not Mondays or Fridays. So was nearly every day something went out.
I’m including bug fixes here but nothing got stalled. We had a working iteration loop of Test -> Develop -> Test -> Review -> QA -> Release
Lots of automated checks, investment in infrastructure and good architecture made this possible. Ironically one of the smallest companies I ever worked for
I'm a current DoD contractor (the coding type) and the agile/waterfall mentality still holds up. Different departments and different projects within each department will have different methodologies for planning and delivering products.
However, where I am, there is a huge emphasis on "getting shit done" well ahead of schedule. I'm lucky to work with smart and motivated folks who together, work well and we are wildly successful.
My biggest gripe, shared among many of my coworkers, is the level of micromanagement. Multiple business analysts, product owners, stakeholders, SCRUM masters, managers, etc. who all share essentially the same job - delegating and scheduling work. It makes every project 10x more difficult and complex than it otherwise should be.
An example just from this past week:
Project A: we are delivering a single product. Three managers, two SCRUM masters, multiple business analysts. We engineers have four different JIRA/task-boards to keep up with. All of them overlap, three of them are essentially the same board in different spaces.
Each day we have 2-4 hours worth of meetings spread across stakeholders amongst those three boards where we regurgitate the exact same information.
It's repetitive, backward, and a time-waste. Some engineers are currently "revolting" by saying what I'm saying now - in the meetings themselves. We've had multiple blowups with management.
Having meetings and discussions spread so wide leads to a lot of confusion because us engineers appear to be the only conduit with which the analysts and managers are communicating. Often times, the different stakeholders will ask for the same features in different meetings, other times they ask for different or conflicting features which leads to confusion and the need to schedule more meetings to clarify as they are not communicating amongst each other.
The issue I found in defence, when your software integrates in to a very large engineering project with a mix of hardware and software is that often times waterfall is appropriate because everything has to be decided upfront where possible.
For tight hardware integration, iteration on design isn't appropriate because the hardware design was locked in to begin manufacturing.
Quite often software life cycles miss the hardware rapid prototyping phase and so there's a desynchronisation of iterative design.
Within the framework of the hardware, yes you can iterate on the specific implementation but the contract with the external hardware and the timeliness to deliver was fixed.
Agile struggles without agile being applied end-to-end, and so you get agile/waterfall mixes.
In my experience, they can still work. In my case we have annually field deployed (and difficult to update) firmware. Firmware feature additions and changes are an up-front known quantity (even if they turn out to be insufficient for the in theater usage). We can't change them. That's the root where any agile approach reaches an external impact limit.
> The problem is that you can't take a process and put software drones on it and deliver software of any complexity. That goes against much (but not all!) of the DoD paradigm, it is interesting to see they might be changing.
My (outside) impression has changed the more I've heard over the years, to the point that I assume the DoD and military are acutely aware that the drones aren't really interchangeable, and that at every level they cultivate people under them they know "get shit done" and hand the important projects to them. Those people in turn have those under them that "get shit done" they hand the important things to, and so on, all the way down to privates in the military. The important thing being that those people that get shit done are given a lot more leeway in how they accomplish things (which might later become doctrine of successful), and this is specifically how those orgs evolve over time.
Again, that's my outside perspective from someone that's interested whenever I hear it and reads/listens to history sources (podcasts,etc).
> to the point that I assume the DoD and military are acutely aware that the drones aren't really interchangeable, and that at every level they cultivate people under them they know "get shit done" and hand the important projects to them.
Within military troops, yes. That's an active goal, especially on the officer side where the whole career path is pointed towards increasing levels of responsibility and leadership. The problem is not in the troops, but in the program offices.
I was on a spectacularly failed Waterfall (big-W, they were explicit about what they were doing) where they handed the task of managing a large, multi-billion dollar software system (from the DOD side) to a 1st Lt history major. She had no preparation for the role. Software acquisition in DOD is, mostly, a clusterfuck. I felt bad for her because she legitimately did not know anything more than the basic legal and financial side of things, she had no technical comprehension and had to trust people (contractors) who were lying about the system status and quality (what was delivered, after she had moved on, was years late and had a fraction of the target capabilities).
Program offices are not generally composed of people who have real-world experience with software development. Remarkably they have great engineers, good finance people, but the software person (often singular, even in programs that are almost exclusively composed of software systems) is usually not from a software background or a straight out of college EE major (more often than not, occasionally a CS major).
> usually not from a software background or a straight out of college EE major (more often than not, occasionally a CS major).
Ugh, I can see that being a problem. I remember my practical experience in projects after graduating with a CS degree, and it was close to non-existent (a class in development methodologies without experiencing it yourself is just a survey of all the things you can do badly if you were to be in charge).
Related to agile on DoD projects, here’s a presentation from Lockheed Martin about their adoption of SAFe (Scaled Agile Framework) for their F-16 software projects. While agilists consider SAFe to be “waterfall in agile clothing” sold by consultants, Lockheed Martin successfully reduced their cycle time from 3-4 years per program increment (delivered to Flight Test team) to 9 months. They’re delivering smaller increments to QA and iterating on feedback more frequently.
> I've heard some software devs state that "the waterfall method was never really used".
What they mean by that is that the person who invented the phrase "waterfall method" did it specifically to tell the world about something that did not work. In fact, even going back to the NATO software engineering conferences of the 1960s, they were discussing how development needs to be agile.
But sure enough, few big organisations got that memo.
DOD codified Royce's waterfall (which he, himself, did not name in that paper) in a set of documents on officially endorsed software acquisitions processes. There were others, but Waterfall was one of them. Its critical failing is that it's an idealized process that cannot withstand uncertainty or change, but it looks really pretty in charts so people selected it without serious thought put into it.
> The problem is that you can't take a process and put software drones on it and deliver software of any complexity.
That would be news to most tech companies, these days, that seem to be obsessed with hiring huge armies of inexperienced engineers, relying on the process to make them magically deliver code like they were masters.
I worked for a couple of major corporations, and got lots of waterfall. A the Japanese company, I quickly learned never to utter the dark majik phrase "A G I L E," as it conjured demons from managers.
Also, looking at that PDF, I realize that it is a slide deck.
Agile had the scrummaster role. The authors of the Agile manifesto said that this role was supposed to rotate in the group of coders and eventually be phased out once the group got used to the process. The agile process in its final form has no managers.
That's Scrum, not Agile. Scrum is a way to, theoretically, get to or introduce Agile concepts but it is not the entirety of Agile (either the theoretical or practical forms).
Also, you can edit your comments which is better than tacking on additional self-replies just a minute or two after.
The Agile manifesto has nothing to do with huge armies of inexperienced code monkeys. On the contrary, it's focused on empowering small, highly competent groups to deliver the most value through quicker feedback loops.
Just out of curiosity, how does someone with in-demand, highly-transferable skills like software development find themselves working in a place they generally dislike for 15 years without leaving sooner?
Oh, I had an awesome time for much of it. Small business with many great friends and we got into many antics over those years through thick and thin. Also, I got to do really cool, nearly constant greenfield work, like flight simulators, robot simulators, radar simulations, robotics, haptics, and then there was some non-DoD work like a virtual gorilla VR experience, gaming stuff, and all kinds of funs times and trips to Game Dev and E3 conferences. I once got to fly on a mid-air refueling mission. Got to see one of the Mars rovers as it was being built. Went on a Naval frigate. Ran a class for some pilot trainees in a building next to the Blue Angels home base.
But there was lots of boring bullshit in there too ;)
As a software developer with 15 years of experience who produces completed software products, I'm wary of this mindset because it's often used as an excuse to tolerate or reward incompetence.
Software can be considered done with respect to any given set of requirements. Now if the requirements change, then of course, the software may need to be updated - but even then, it should be designed in such a way that the impact of requirement changes are minimal. Software should, to some extent, anticipate requirement changes. Of course this is not easy, but to claim that it's impossible is to admit defeat. You need highly skilled people making those architectural decisions.
Some software is subjected to constantly changing requirements and these can be said to never be done but that's far from being the case for all software. Libraries and modules are software too and these should be generic enough to handle requirement changes without code changes since they should not concern themselves with the business domain.
I have completed open source projects which did not need to be updated in years and worked just as well. I've worked on other projects which had to be updated because of vulnerabilities found in dependencies but these did not require any code changes on my end.
I consider it a huge red flag if some library was working fine for 5 years and then people decided to change its interface in a non-trivial way.
I was tolerant when async/await was rolled out to major programming languages; this affected and continues to affect dependencies but it's a meaningful conceptual improvement and helps with future maintainability. I hope that programming standards organizations will slow down after that in terms of changes which affect function interfaces because these decisions have huge flow-on impacts.
It's totally possible to produce completed software (especially libraries and frameworks) but the industry has to aim for it and expect it.
The current goal of industry seems to be job creation, not the creation of robust working products. Accepting that software can never be done right from the beginning is going to be a disaster. It should only be tolerated in the context of requirement changes.
As a fellow software developer, I respectfully disagree.
While a piece of software may satisfy its requirements, it will never satisfy them perfectly or optimally. All software contains bugs, unexpected failure modes, and undiscovered security vulnerabilities. I am convinced that there are essentially zero exceptions to these claims.
Furthermore, the environment in which software runs must change over time. This causes "bit rot" as (say) dependencies stop receiving security updates or platforms become unsupported.
Finally, there's the human factor. When software runs on old technologies, it will become progressively harder for new developers to understand. There will consequently often be a steady decline in available talent.
For these reasons, even if the requirements of a piece of software never change, it will always require ongoing maintenance to keep it running, much like a piece of physical infrastructure.
If performance requirements change, then that would count as a change of requirements in my books. It should be possible to decide ahead what an acceptable target is. Also, it's not going to affect every part of the software.
About changes in tech and environments. It's possible to build portable software. Whether or not software needs to be portable should be treated as a requirement. I disagree with any argument which suggests that it's not possible to foresee this.
It's not always the case that old technologies are harder to understand. This is mostly because software is still a relatively new area in the grand scheme of things. There are some old projects which I find easier to understand. Especially those built with languages which stood the test of time like C/C++, Java, Python, JavaScript...
Requirements aren't from God. They're just a tool for facilitating transactions with contractors. Whatever the software does or is used for, can almost always be optimized further.
Yes and that’s historically a popular way of organizing software development, particularly in enterprise IT. In the more forward-looking parts of the tech world, we tend to see software teams charged directly with the success of their software in the world. Engineers are expected to both come up with and deliver on software changes that advance business goals. Stakeholders participate in shaping and prioritizing, but ultimately you are there to solve their problems, not build to their specs. IMO a better system!
> All software contains bugs, unexpected failure modes, and undiscovered security vulnerabilities. I am convinced that there are essentially zero exceptions to these claims.
/bin/true (humbly submitted as a non-zero exception to those claims)
For every program I have ever worked on, the "finished state" is when the tool reads your brain, does all the work for you, and transfers the resulting dollars in to your bank account. Until then, we always have work to do that brings us closer.
Imagine Photoshop, they continually add new tools which let you get the intended results with less manual work. The end state of photoshop is you just open the file and it immediately shows you the finished picture you wanted without having to do anything. We are decades away from that.
What if a requirement is “system is secure from cyber attackers”?
I had a discussion the other day about how to tell when a web service can really be considered secure enough for somebody important to store sensitive data on (Eg, medical software); when do you say “the requirement has been met”? Does it become un-met next week when a new exploit is discovered in a package you rely on?
I want to read more of this later. After 4 years in DoD contracting, at a quick glance, some of this hits the nail right on the head:
>Software is made by people and for people, so digital talent matters.
The company I was at did not acknowledge this. To them, engineering hours were fungible. Two weeks from an engineer who is doing the bare minimum and coasting to retirement (in DoD contracting, there are lots of these type) is the same as a mid-level engineer who is up-to-date on the latest technology and working their hardest to push the project forward. The latter is usually paid less than half the former in the industry.
>Software is different than hardware (and not all software is the same).
What works in other industries doesn't always work in defense. It's a unique space. CI? Good luck running automated "intercept a hypersonic missile" tests. This doesn't get enough attention. However, there is a lot that can be done in pure software testing, even for DoD applications. They view it too much like hardware development and that hamstrings their software efforts.
The bigger issue that is industry-wide that I noticed in my time there, is that the incentive system in the industry is broken. Contractors take an educated guess up front on what it will cost them to build a system for the government, and the government chooses from those bids, weighing capabilities, cost schedule, etc. The contractor who wins, then tracks their progress to completion by how many hours they have worked on the contract, relative to their total. This is how they get paid. For instance, if it is predicted to take 100 hours to deliver a system, and that system is $1 million, each hour would earn the contractor something like $10,000 (obviously there are other non-labor costs and this is a hand waving simplification, but to an extent is true). So, there is limited (bonuses that are tiny relative to the value of the contract) incentive to come in on time. There is a great incentive to over-hire, pay staff to sit around doing nothing, and collect money for it. This disincentivizes innovation, because you don't want to have a predicted cost that is astonishingly low for your next contract; you want it to be just low enough to undercut your next competitor. That's how you maximize profits, and us taxpayers get a terrible deal.
>There is a great incentive to over-hire, pay staff to sit around doing nothing, and collect money for it.
Oh god. Flashbacks to my time as a contractor to Aus DoD. Over 18 months I produced a small handful of xml schemas, a single java class that called out to their approved FTP library, and ate enough doughnuts to put on 20kg.
There are multiple contract types, and you're correct that the "cost-plus" paradigm (where the contract covers the contractor's costs plus a profit margin) has some perverse incentives. I saw a number of smaller efforts at Medicare use the "firm fixed price" paradigm (where the total cost of the contract is agreed to up front), which allows the contractor to keep whatever they save by delivering early or using a smaller than predicted staff but makes the contractor liable for any cost or staffing overages. That system seemed to produce better aligned incentives, but it's more difficult to apply to larger projects.
Yeah, there is a push to move to the firm fixed price contracts. The contractors hate those, since they’re higher risk. Both parties would benefit from something more incremental and agile. For example, maybe 10% of the contract is paid at specific, agreed-upon milestones. That would allow for better incentives at lower risk to the contractor, since forecasting a decade-long contract is immensely difficult.
I work in the cleared DoD contracting space with my own small biz. If you’re even a little better than those other guys just sitting around doing nothing, you should go out on your own as a 1099.
Feel free to ping me dale@1099fedhub.com if you’re curious about going down that path.
I know it's not what they mean and I agree with the general idea.
But for very small very well contained libraries I would argue that they sometime _are_ done (for many years, not forever; in a sense that any changes you can come up with add no relevant value at all, including no relevant perf. improvements or better maintainability).
At least if written in a language with nice forward and backward compatibility, cross platform support and a stable "base" ecosystem.
(So not C,C++ due to not default package manager/build system and too much divergence between platforms (e.g. unsigned long), not JS because their core build tooling changes to much (through it's getting better!), etc.)
> As stated by McIlroy, and generally accepted throughout the Unix community, Unix programs have always been expected to follow the concept of DOTADIW, or "Do One Thing And Do It Well." There are limited sources for the acronym DOTADIW on the Internet, but it is discussed at length during the development and packaging of new operating systems, especially in the Linux community.
> As Steve Jobs observed, 8 one of the major differences between hardware and software is that for hardware the “dynamic range” (ratio between the best in class and average performance) is, at most, 2:1. But, the difference between the best software developer and an average software developer can be 50:1, or even 100:1, and putting great developers on a team with other great developers amplifies this effect. Today, in DoD and the industrial base that supports it, the people with the necessary skills exist, but instead of taking advantage of their skills we put them in environments where it is difficult for them to be effective. DoD does not take advantage of already existing military and civilian personnel expertise by offering pay bonuses, career paths that provide the ability to stay in their specialization, or access to early promotions. Skilled software engineers and the related specialties that are part of the overall software ecosystem need to be treated as a special force; the United States must harness their talent for the great benefits that it can provide.
As if we software people didn't have overinflated egos already...
This DoD report talks about 100x developers existing (and of course HNers imagine ourselves towards that better-than-average end of the scale).
The DoD here even uses the (lowercase) term "special force", which sounds like we're not nerd typists, but super-cool elite operators.
Ironically I suspect that wind l while the idea will feed many egos the reality would crush or frustrate those same people. If the DoD certifies someone as a 100x developer it would be after crafting a ruthlessly banal evaluation system that would likely lack the romantic attitude towards software many developers have in my experience. Especially if the criteria included some social skills component that determined part of a developers rating by how well they synergize with similarly rated peers.
I think the DoD has had many outstanding R&D successes, hiring or contracting elite engineers and scientists to do amazing things, which beat most of what we see from tech private sector B2C and B2B businesses.
That's one place you'll need some people who can find brilliant solutions or insights that most people simply cannot, no matter whether you throw 100x times the number of people at the problem.
But if they're having trouble attracting and retaining as many 1x-3x everyday software developers as they'd like, I agree, it'd be easy to imagine an attempt that fails miserably due to stereotypical bureaucracy.
Paying close to FAANG TC levels would be a start, since it would cover the monetary factor, and i think the prestige factor would quickly follow. (I bet few people think adtech, fintech, or yet-another-microblogging/messaging/profile-app are actually cool anymore -- only the money is what makes those companies prestigious.)
Maybe pay 80% of FAANG, so the FAANGs will still get the people who only want money, and you offer better non-monetary value to make up that 20%. (80% should be close enough they can afford to live in the same neighborhoods and send their kids to the same schools as FAANG, rather than be lower class that could upgrade just by grinding Leetcode for a few months.)
Then manage those units like they are very serious excellent product development organizations where everyone wants to work. And figure out how to do the culture better than all the companies that got too much into class thinking, and too much jumping through the hoops for interview and pursuit of promotions, rather than focus on the mission and team.
I feel Scrum has evolved into something contradicting the original Agile practices.
These were more of a template on how to develop, not a concrete recipe. Scrum then established a process where there was need for a family of processes, continuously evolving, ever changing.
The very fact that "you absolutely have to have a specific ceremony (i.e.: Retro when no topics)" is a red flag which imho puts the cart before the horse.
Why is there a "Scrum Master" anyway, if individuals are encouraged to interact and own the process? Suppose an individual wants to change the process, and reaches consensus within the team/community. Why would a Scrum Master / Manager even be able to veto any decision?
Why is there a Sprint when SW development is actually a marathon? Uncle Bob talks about this in several of his Clean Methodology books.
I don't think that Agile is dead [1], although it has been taken over by people eager to sell you a *cough* certification (Scrum or any other). It's not dead because its genius is that it's constantly evolving, and you can always interpret it differently since it's only a template. Scrum, on the other hand.. well..
For the future, I look forward to not putting an equal sign between Scrum and Agile, even though apparently Scrum is how you "do" Agile, according to the industry.
The problem isn't that DoD requires waterfall for software. The problem is the Planning Programming Budgeting Execution cycle. If you solve a problem, you have at least an 18 month hole in the schedule. This drives PMs to waterfall so the can project funding out over the POM hole. It's profoundly broken and kept that way by Congress critters in the pocket of the prime integrators.
A question I have: do people working on DoD software ever get to use it themselves? I assume many of them don't (if they're working on, say, a weapons system).
This situation often (IMHO) makes it much harder to deliver high-quality software. No amount of surface-level UI testing replaces the knowledge you get from actually using a system yourself.
The 10 recommendations found on the "cheat sheet" (page xv):
Line of Effort A (Congress and OSD):
Refactor statutes, regulations, and processes for software
* [A1] Establish one or more new acquisition pathways for software that prioritize continuous integration and delivery of working software in a secure manner, with continuous oversight from automated analytics
* [A2] Create a new appropriation category for software capability delivery that allows (relevant types of) software to be funded as a single budget item, with no separation between RDT&E, production, and sustainment
Line of Effort B (OSD and Services):
Create and maintain cross-program/cross-Service digital infrastructure
* [B1] Establish and maintain digital infrastructure within each Service or Agency that enables rapid deployment of secure software to the field, and incentivize its use by contractors
* [B2] Create, implement, support, and use fully automatable approaches to testing and evaluation (T&E), including security, that allow high-confidence distribution of software to the field on an iterative basis
* [B3] Create a mechanism for Authorization to Operate (ATO) reciprocity within and between programs, Services, and other DoD agencies to enable sharing of software platforms, components, and infrastructure and rapid integration of capabilities across (hardware) platforms, (weapon) systems, and Services
Line of Effort C (Services and OSD):
Create new paths for digital talent (especially internal talent)
* [C1] Create software development units in each Service consisting of military and civilian personnel who develop and deploy software to the field using DevSecOps practices
* [C2] Expand the use of (specialized) training programs for CIOs, SAEs, PEOs, and PMs that provide (hands-on) insight into modern software development (e.g., Agile, DevOps, DevSecOps) and the authorities available to enable rapid acquisition of software
Line of Effort D (DoD and industry):
Change the practice of how software is procured and developed
* [D1] Require access to source code, software frameworks, and development toolchains —- with appropriate IP rights —- for DoD-specific code, enabling full security testing and rebuilding of binaries from source
* [D2] Make security a first-order consideration for all software-intensive systems, recognizing that security-at-the-perimeter is not enough
* [D3] Shift from the use of rigid lists of requirements for software programs to a list of desired features and required interfaces/characteristics to avoid requirements creep, overly ambitious requirements, and program delays
The attitude toward the development process expressed here is not uniformly accepted in all programming subcultures. There is a major cultural divide between embedded and server/desktop programming. My example is very old, but I had occasion to discuss software development with embedded systems engineers from Air Force projects in the pre-Ada days. They were all vehemently against the idea of software maintenance. They felt that their job was to deliver a "correct" implementation of a fully specified system.
That model makes sense for aircraft, in my opinion. The software isn’t a standalone product that can be independently continuously improved. It’s tightly integrated with safety critical physical systems. You need to do the upfront work of specifying how the software and hardware interact and do the complex systems safety analysis. Any change to the software specification after design can have unintended effects on the overall behavior of the aircraft.
> In a year replete with reports on artificial intelligence, quantum computing, and blockchain, may seem mundane to write about software, but software is the foundation of all things digital, and the Department does it exceedingly poorly.
I take issue with the title. Software is constantly "done" and then new requirements arrive, causing a new version of the completed software to be built. A suspension bridge is considered "done" even if it's getting continuously repainted.
You are thinking in terms of software in the non-government world. Contractors try really hard never to finish software so they can keep extending their contracts. This document is about the military world, and if you don't have experience in it, it can be hard to interpret!
Look at how the TV series industry is hard trying with cliffhangers or story telling arcs without a closure...
This issue runs much deeper:
It is how the economy operates: you need a roof to protect you, every day, you need to eat, every day, until you die in decades. Many economic activities, if done honestly and fairly, cannot justify a permanent income.
What is meant by done? As part of system it co-evolves with it and due with it. It dies with the system it sort of “parasite” on or it is the key part of the system or it is the system. Still evolve until death.
It's like reading a report by Darth Vader about the "national defense" of the galactic empire, explaining what's necessary for crushing "our adversaries" the rebels.
I was a DoD contractor (the coding type, not the shooting type) for almost 15 years, and this is a really interesting read. I've heard some software devs state that "the waterfall method was never really used". Those devs were not DoD software contractors for almost 15 years. DoD was and is all about the waterfall model. I have spreadsheet PTSD from seeing so many waterfall diagrams and project manager's Gantt charts. I was working on a peanut-size project for the Navy (around $5-10million over 5 years) when Agile began picking up steam (mid 00s). I introduced the team to it and we began doing Agile to decent effect (at least in comparison to waterfall), and introduced it to our Navy sponsors. I remember some higher-ranking individual ask, "wait, so this method can actually deliver software?" Or something like that. Of course, Agile is no panacea, but the iterative nature was waaaay better than what the DoD was doing.
I really like how they talk about "individuals of talent" in the document. Much of the DoD focuses around process, to great effect. The problem is that you can't take a process and put software drones on it and deliver software of any complexity. That goes against much (but not all!) of the DoD paradigm, it is interesting to see they might be changing.
Of course, they will meet great resistance, after all, the first job of a successful contractor is to extract as much money as possible from the government with as little work as possible. "Never finish a contract" was the mantra we had learned from the giant contractors.
Glad to be out of there, but it was a really interesting experience!
I keep banging on about "software is a new form of literacy".
Imagine you have a group of illiterate managers who have just hired a group of writers to write some company policy documents.
You could have the managers (carve / draw / modern dance) their requirements in a waterfall manner.
Of course you could get one of the managers to sit down with the developers and tell them how the policy's work each day, embed with them Agile style.
However in the end you can't be everywhere, sometimes there aren't enough people in the company who understand what is needed, or even be sure that the embedded people agree on what is needed.
These are business problems - but they are also literacy problems. We have a literate society and so there are enough literate people to write policy documents to cover a whole compmay.
We need to make enough people who can code, so they can write the policy documents of the future.
This is an interesting take. I've always rolled my eyes at the people who say "everyone needs to know programming" but when you put it like this, yeah, why not?
The only thing I'm not convinced about is whether programming is something of a functional specialisation or if it will (GAS already?) become a literacy-like thing.
I mean, we generally think of numeracy as something widespread in the West for the past few decades, but the average person really does not do even basic arithmetic very well. Are we really going to do a better job of teaching people programming, or will we have to resign to the reality that most people won't learn it well?
My take on this is the basics of astronomy. The history of how astronomy slowly worked out the earth goes round the sun, distances etc is fascinating (#)
The bit that's gets me is how fast does earth spin (without which sun rise is a bit hard to explain). Until it was accepted that air was just a thin atmospheric layer, it becomes insane to imagine we spin at a 1,000 miles an hour though a air medium that extends to Jupiter. How do you keep your newspaper still.
Kids today have Mickey Mouse cartoons explaining space rockets, moon to mars, earth spinning etc. We just accept it as true without trying to be clever about it
The same is true for literacy and numeracy - just having a society where you absorb character arcs, sub clauses, chapters, publishing, versions etc makes so much difference. Mathematical symbols were the names of Ed Sheeran albums.
Look at something simple like version control - the idea is fairly simple and hot is almost perfect for it, but oh god have you tried to use Sharepoint. Once you are in the WYSIWYG mind set, version control breaks horribly and most most business users should have this all the time but it's not in their world view. Explain it to them and they say "but how does your nreasoaper not flap around in thousand mile hour winds"
(#) Look for episode on the podcast "Subject to chnage"
I've been harping on this for decades. Back in the early 2000s there was STILL business news interviews with some CEO bragging about how he has his secretary print out the emails and doesn't know how to use a computer. Thankfully that ended probably around 2005.
Fully 25 years after computers-in-business, especially for any significant size of business, was virtually universal.
Humans don't manage businesses anymore. They are essentially all software managed. Business management is now really meta-management of business management software.
Yes!
>>> Business management is now really meta-management of business management software.
It really ought to look like :
for each emp in employees_by_dept('accounts'): emp.getvacationplaennedthismonth
or something
I haven't read the doc, but I worked on lawsuits (I'm a lawyer) that at their core were asserting that agile software development isn't as safe or bug proof as waterfall. Basically in the medical device (or adjacent) space the idea asserted was that 'fail fast' was illegal and didn't have enough safety checks.
Agile software development is a tough pill to swallow for non-engineers. The lawyers and business people can't wrap their head around the idea that building iteratively could actually reduce bugs and improve quality - because the way they work is via process planning docs and checklists.
Such a great point.
I think one way to go about it is to just put the Agile bits inside a Waterfall.
Which is frankly how it works anyhow.
You can characterize early sprints at 'research'.
Later the sprints are just 'sub projects'.
I think people could be convinced that 'feature refinement' is a real thing. Put it out in the field, doesn't work the way we want it, and so it needs to be refined.
I know that I would hide 'Agile' to anyone outside the team and just talk in simple, easier to understand phases.
It's also possible to put checklists in those phases, and if you say someting like '75% features' at such and such phase and assume '25% of that 75% will be reworked' it's not irrational either.
But you are right, culture matters.
Building iteratively can reduce bugs and improve quality other things being equal. The problem with how Agile methods sometimes get implemented in practice is that an excess focus on quick-and-dirty iteration to the exclusion of everything else leads to having mountains of very poor quality code, with little to no support structure around it in the form of broader design or documentation. Agile methods advocate for growing these kinds of support structures via refactoring, but this is often not done to anywhere near an adequate extent.
I think this is true even in “civilian” industries. Can say doing this for over a decade now that I have worked at exactly one place that followed anything close to what was laid out in the Agile manifesto.
Every other place always sings the praises of agile and/or scrum but it’s always just waterfall with ceremonies.
I feel like the collective of how software is developed evolved little from circa 2006
The notion that gets talked about at Netflix is “context not control” and I think it would go along way if everyone looked at software development management this way. Too many people try to control software development lifecycles instead of just providing engineers enough context to get their work done and trusting that reasonable people will make reasonable decisions
Large organizations value predictability.
IMO it’s not this simple. Culture has more to do with it than anything. See my note about Netflix having this mantra of “context not control”. Managers are less inclined to be domineering in that setting and allow people to do the work in a reasonable way because they are given the context of why their work is being done in the first place.
Not perfect but a more than above average way to approach the problem yet this kind of thinking or its permutations have not caught on at all. It’s still top down process oriented models
Agile is a tough pill to swallow for many engineers, but it's become part of the theology of software development, so everyone's afraid to object to it.
This is so true. They don’t teach agile in management classes and ‘planning’ generally means years, not t-shirt sizes. I believe that also many economic theories and practices don’t regard agile practices. I’ve always felt that the business and their planning are at odds with the agile ways.
> management classes
It would seem they don’t do a very good job teaching how to think about uncertainty, risk, and unknowns, which is what agile is good at addressing. If you want a suit to lose their mind, tell them they can only pick two of feature, quality, or schedule.
Waterfall seems a lot better when you have done something before and there's nothing to discover.
> Waterfall seems a lot better when you have done something before and there's nothing to discover.
I've said this here on HN before, but the only Waterfall projects I've seen it work reliably were where:
1. The systems were small (in which case any method would work)
2. The systems were trivial (in which case any method would work)
3. The systems were well-understood by the people making them (in which case any method would work)
It does not handle large, non-trivial, or novel systems very well. Almost without fail they will deliver late, over budget, or deliver the wrong thing. This can be a win for contractors, though, who get to bid on the rework or sustainment contract for their own work.
What a bizarre claim. Agile doesn't mean "cowboy coding" (I've met as many cowboy coders in Waterfall, may it die a fiery death, as Agile projects).
Nothing about Agile says, "Don't ever test and validate this system before deploying." Anyone who believes that is, frankly, a moron.
In my experience it was less a lack of intelligence and more results-oriented thinking (they thought we were wrong so they found a way to argue it).
If you have twenty versions of a thing and multiple deployments to fix and fine tune it, that means to this person there are 20 plus examples of "mistakes" you made in building it. They were looking for evidence that the software was built with "poor quality controls" and ironically interpreted iteration as evidence of poor build quality.
The ONLY way to get quality is through quantity. That's the key insight.
"Breaks when you try to use it" is a quality that I can usually achieve on the first try.
Or did you mean some quality in particular?
Quantity of... dollars?
I’m betting releases and iteration cycles they’re tied to. The most agile place I ever worked shipped code most days of the week in some fashion. For control reasons we released Tuesday-Thursday not Mondays or Fridays. So was nearly every day something went out.
I’m including bug fixes here but nothing got stalled. We had a working iteration loop of Test -> Develop -> Test -> Review -> QA -> Release
Lots of automated checks, investment in infrastructure and good architecture made this possible. Ironically one of the smallest companies I ever worked for
Iterations.
I'm a current DoD contractor (the coding type) and the agile/waterfall mentality still holds up. Different departments and different projects within each department will have different methodologies for planning and delivering products.
However, where I am, there is a huge emphasis on "getting shit done" well ahead of schedule. I'm lucky to work with smart and motivated folks who together, work well and we are wildly successful.
My biggest gripe, shared among many of my coworkers, is the level of micromanagement. Multiple business analysts, product owners, stakeholders, SCRUM masters, managers, etc. who all share essentially the same job - delegating and scheduling work. It makes every project 10x more difficult and complex than it otherwise should be.
An example just from this past week: Project A: we are delivering a single product. Three managers, two SCRUM masters, multiple business analysts. We engineers have four different JIRA/task-boards to keep up with. All of them overlap, three of them are essentially the same board in different spaces. Each day we have 2-4 hours worth of meetings spread across stakeholders amongst those three boards where we regurgitate the exact same information.
It's repetitive, backward, and a time-waste. Some engineers are currently "revolting" by saying what I'm saying now - in the meetings themselves. We've had multiple blowups with management.
Having meetings and discussions spread so wide leads to a lot of confusion because us engineers appear to be the only conduit with which the analysts and managers are communicating. Often times, the different stakeholders will ask for the same features in different meetings, other times they ask for different or conflicting features which leads to confusion and the need to schedule more meetings to clarify as they are not communicating amongst each other.
The issue I found in defence, when your software integrates in to a very large engineering project with a mix of hardware and software is that often times waterfall is appropriate because everything has to be decided upfront where possible.
For tight hardware integration, iteration on design isn't appropriate because the hardware design was locked in to begin manufacturing.
Quite often software life cycles miss the hardware rapid prototyping phase and so there's a desynchronisation of iterative design.
Within the framework of the hardware, yes you can iterate on the specific implementation but the contract with the external hardware and the timeliness to deliver was fixed.
Agile struggles without agile being applied end-to-end, and so you get agile/waterfall mixes.
In my experience, they can still work. In my case we have annually field deployed (and difficult to update) firmware. Firmware feature additions and changes are an up-front known quantity (even if they turn out to be insufficient for the in theater usage). We can't change them. That's the root where any agile approach reaches an external impact limit.
Our little bubble is agile in a waterfall world.
> The problem is that you can't take a process and put software drones on it and deliver software of any complexity. That goes against much (but not all!) of the DoD paradigm, it is interesting to see they might be changing.
My (outside) impression has changed the more I've heard over the years, to the point that I assume the DoD and military are acutely aware that the drones aren't really interchangeable, and that at every level they cultivate people under them they know "get shit done" and hand the important projects to them. Those people in turn have those under them that "get shit done" they hand the important things to, and so on, all the way down to privates in the military. The important thing being that those people that get shit done are given a lot more leeway in how they accomplish things (which might later become doctrine of successful), and this is specifically how those orgs evolve over time.
Again, that's my outside perspective from someone that's interested whenever I hear it and reads/listens to history sources (podcasts,etc).
> to the point that I assume the DoD and military are acutely aware that the drones aren't really interchangeable, and that at every level they cultivate people under them they know "get shit done" and hand the important projects to them.
Within military troops, yes. That's an active goal, especially on the officer side where the whole career path is pointed towards increasing levels of responsibility and leadership. The problem is not in the troops, but in the program offices.
I was on a spectacularly failed Waterfall (big-W, they were explicit about what they were doing) where they handed the task of managing a large, multi-billion dollar software system (from the DOD side) to a 1st Lt history major. She had no preparation for the role. Software acquisition in DOD is, mostly, a clusterfuck. I felt bad for her because she legitimately did not know anything more than the basic legal and financial side of things, she had no technical comprehension and had to trust people (contractors) who were lying about the system status and quality (what was delivered, after she had moved on, was years late and had a fraction of the target capabilities).
Program offices are not generally composed of people who have real-world experience with software development. Remarkably they have great engineers, good finance people, but the software person (often singular, even in programs that are almost exclusively composed of software systems) is usually not from a software background or a straight out of college EE major (more often than not, occasionally a CS major).
> usually not from a software background or a straight out of college EE major (more often than not, occasionally a CS major).
Ugh, I can see that being a problem. I remember my practical experience in projects after graduating with a CS degree, and it was close to non-existent (a class in development methodologies without experiencing it yourself is just a survey of all the things you can do badly if you were to be in charge).
Related to agile on DoD projects, here’s a presentation from Lockheed Martin about their adoption of SAFe (Scaled Agile Framework) for their F-16 software projects. While agilists consider SAFe to be “waterfall in agile clothing” sold by consultants, Lockheed Martin successfully reduced their cycle time from 3-4 years per program increment (delivered to Flight Test team) to 9 months. They’re delivering smaller increments to QA and iterating on feedback more frequently.
https://youtu.be/nvfYLZ52zX0
> I've heard some software devs state that "the waterfall method was never really used".
What they mean by that is that the person who invented the phrase "waterfall method" did it specifically to tell the world about something that did not work. In fact, even going back to the NATO software engineering conferences of the 1960s, they were discussing how development needs to be agile.
But sure enough, few big organisations got that memo.
DOD codified Royce's waterfall (which he, himself, did not name in that paper) in a set of documents on officially endorsed software acquisitions processes. There were others, but Waterfall was one of them. Its critical failing is that it's an idealized process that cannot withstand uncertainty or change, but it looks really pretty in charts so people selected it without serious thought put into it.
> The problem is that you can't take a process and put software drones on it and deliver software of any complexity.
That would be news to most tech companies, these days, that seem to be obsessed with hiring huge armies of inexperienced engineers, relying on the process to make them magically deliver code like they were masters.
I worked for a couple of major corporations, and got lots of waterfall. A the Japanese company, I quickly learned never to utter the dark majik phrase "A G I L E," as it conjured demons from managers.
Also, looking at that PDF, I realize that it is a slide deck.
That presentation must have been a soporific.
Agile had the scrummaster role. The authors of the Agile manifesto said that this role was supposed to rotate in the group of coders and eventually be phased out once the group got used to the process. The agile process in its final form has no managers.
That's Scrum, not Agile. Scrum is a way to, theoretically, get to or introduce Agile concepts but it is not the entirety of Agile (either the theoretical or practical forms).
Also, you can edit your comments which is better than tacking on additional self-replies just a minute or two after.
https://m.youtube.com/watch?v=hG4LH6P8Syk at 21:10
The Agile manifesto has nothing to do with huge armies of inexperienced code monkeys. On the contrary, it's focused on empowering small, highly competent groups to deliver the most value through quicker feedback loops.
Just out of curiosity, how does someone with in-demand, highly-transferable skills like software development find themselves working in a place they generally dislike for 15 years without leaving sooner?
Oh, I had an awesome time for much of it. Small business with many great friends and we got into many antics over those years through thick and thin. Also, I got to do really cool, nearly constant greenfield work, like flight simulators, robot simulators, radar simulations, robotics, haptics, and then there was some non-DoD work like a virtual gorilla VR experience, gaming stuff, and all kinds of funs times and trips to Game Dev and E3 conferences. I once got to fly on a mid-air refueling mission. Got to see one of the Mars rovers as it was being built. Went on a Naval frigate. Ran a class for some pilot trainees in a building next to the Blue Angels home base.
But there was lots of boring bullshit in there too ;)
As a software developer with 15 years of experience who produces completed software products, I'm wary of this mindset because it's often used as an excuse to tolerate or reward incompetence.
Software can be considered done with respect to any given set of requirements. Now if the requirements change, then of course, the software may need to be updated - but even then, it should be designed in such a way that the impact of requirement changes are minimal. Software should, to some extent, anticipate requirement changes. Of course this is not easy, but to claim that it's impossible is to admit defeat. You need highly skilled people making those architectural decisions.
Some software is subjected to constantly changing requirements and these can be said to never be done but that's far from being the case for all software. Libraries and modules are software too and these should be generic enough to handle requirement changes without code changes since they should not concern themselves with the business domain.
I have completed open source projects which did not need to be updated in years and worked just as well. I've worked on other projects which had to be updated because of vulnerabilities found in dependencies but these did not require any code changes on my end.
I consider it a huge red flag if some library was working fine for 5 years and then people decided to change its interface in a non-trivial way.
I was tolerant when async/await was rolled out to major programming languages; this affected and continues to affect dependencies but it's a meaningful conceptual improvement and helps with future maintainability. I hope that programming standards organizations will slow down after that in terms of changes which affect function interfaces because these decisions have huge flow-on impacts.
It's totally possible to produce completed software (especially libraries and frameworks) but the industry has to aim for it and expect it.
The current goal of industry seems to be job creation, not the creation of robust working products. Accepting that software can never be done right from the beginning is going to be a disaster. It should only be tolerated in the context of requirement changes.
As a fellow software developer, I respectfully disagree.
While a piece of software may satisfy its requirements, it will never satisfy them perfectly or optimally. All software contains bugs, unexpected failure modes, and undiscovered security vulnerabilities. I am convinced that there are essentially zero exceptions to these claims.
Furthermore, the environment in which software runs must change over time. This causes "bit rot" as (say) dependencies stop receiving security updates or platforms become unsupported.
Finally, there's the human factor. When software runs on old technologies, it will become progressively harder for new developers to understand. There will consequently often be a steady decline in available talent.
For these reasons, even if the requirements of a piece of software never change, it will always require ongoing maintenance to keep it running, much like a piece of physical infrastructure.
(edit: typo)
If performance requirements change, then that would count as a change of requirements in my books. It should be possible to decide ahead what an acceptable target is. Also, it's not going to affect every part of the software.
About changes in tech and environments. It's possible to build portable software. Whether or not software needs to be portable should be treated as a requirement. I disagree with any argument which suggests that it's not possible to foresee this.
It's not always the case that old technologies are harder to understand. This is mostly because software is still a relatively new area in the grand scheme of things. There are some old projects which I find easier to understand. Especially those built with languages which stood the test of time like C/C++, Java, Python, JavaScript...
Requirements aren't from God. They're just a tool for facilitating transactions with contractors. Whatever the software does or is used for, can almost always be optimized further.
My approach to software development is that the stakeholder is my God and their Requirements are my Commandments.
Yes and that’s historically a popular way of organizing software development, particularly in enterprise IT. In the more forward-looking parts of the tech world, we tend to see software teams charged directly with the success of their software in the world. Engineers are expected to both come up with and deliver on software changes that advance business goals. Stakeholders participate in shaping and prioritizing, but ultimately you are there to solve their problems, not build to their specs. IMO a better system!
> All software contains bugs, unexpected failure modes, and undiscovered security vulnerabilities. I am convinced that there are essentially zero exceptions to these claims.
/bin/true (humbly submitted as a non-zero exception to those claims)
For every program I have ever worked on, the "finished state" is when the tool reads your brain, does all the work for you, and transfers the resulting dollars in to your bank account. Until then, we always have work to do that brings us closer.
Imagine Photoshop, they continually add new tools which let you get the intended results with less manual work. The end state of photoshop is you just open the file and it immediately shows you the finished picture you wanted without having to do anything. We are decades away from that.
What if a requirement is “system is secure from cyber attackers”?
I had a discussion the other day about how to tell when a web service can really be considered secure enough for somebody important to store sensitive data on (Eg, medical software); when do you say “the requirement has been met”? Does it become un-met next week when a new exploit is discovered in a package you rely on?
I want to read more of this later. After 4 years in DoD contracting, at a quick glance, some of this hits the nail right on the head:
>Software is made by people and for people, so digital talent matters. The company I was at did not acknowledge this. To them, engineering hours were fungible. Two weeks from an engineer who is doing the bare minimum and coasting to retirement (in DoD contracting, there are lots of these type) is the same as a mid-level engineer who is up-to-date on the latest technology and working their hardest to push the project forward. The latter is usually paid less than half the former in the industry.
>Software is different than hardware (and not all software is the same). What works in other industries doesn't always work in defense. It's a unique space. CI? Good luck running automated "intercept a hypersonic missile" tests. This doesn't get enough attention. However, there is a lot that can be done in pure software testing, even for DoD applications. They view it too much like hardware development and that hamstrings their software efforts.
The bigger issue that is industry-wide that I noticed in my time there, is that the incentive system in the industry is broken. Contractors take an educated guess up front on what it will cost them to build a system for the government, and the government chooses from those bids, weighing capabilities, cost schedule, etc. The contractor who wins, then tracks their progress to completion by how many hours they have worked on the contract, relative to their total. This is how they get paid. For instance, if it is predicted to take 100 hours to deliver a system, and that system is $1 million, each hour would earn the contractor something like $10,000 (obviously there are other non-labor costs and this is a hand waving simplification, but to an extent is true). So, there is limited (bonuses that are tiny relative to the value of the contract) incentive to come in on time. There is a great incentive to over-hire, pay staff to sit around doing nothing, and collect money for it. This disincentivizes innovation, because you don't want to have a predicted cost that is astonishingly low for your next contract; you want it to be just low enough to undercut your next competitor. That's how you maximize profits, and us taxpayers get a terrible deal.
>There is a great incentive to over-hire, pay staff to sit around doing nothing, and collect money for it.
Oh god. Flashbacks to my time as a contractor to Aus DoD. Over 18 months I produced a small handful of xml schemas, a single java class that called out to their approved FTP library, and ate enough doughnuts to put on 20kg.
There are multiple contract types, and you're correct that the "cost-plus" paradigm (where the contract covers the contractor's costs plus a profit margin) has some perverse incentives. I saw a number of smaller efforts at Medicare use the "firm fixed price" paradigm (where the total cost of the contract is agreed to up front), which allows the contractor to keep whatever they save by delivering early or using a smaller than predicted staff but makes the contractor liable for any cost or staffing overages. That system seemed to produce better aligned incentives, but it's more difficult to apply to larger projects.
Yeah, there is a push to move to the firm fixed price contracts. The contractors hate those, since they’re higher risk. Both parties would benefit from something more incremental and agile. For example, maybe 10% of the contract is paid at specific, agreed-upon milestones. That would allow for better incentives at lower risk to the contractor, since forecasting a decade-long contract is immensely difficult.
I work in the cleared DoD contracting space with my own small biz. If you’re even a little better than those other guys just sitting around doing nothing, you should go out on your own as a 1099.
Feel free to ping me dale@1099fedhub.com if you’re curious about going down that path.
I know it's not what they mean and I agree with the general idea.
But for very small very well contained libraries I would argue that they sometime _are_ done (for many years, not forever; in a sense that any changes you can come up with add no relevant value at all, including no relevant perf. improvements or better maintainability).
At least if written in a language with nice forward and backward compatibility, cross platform support and a stable "base" ecosystem.
(So not C,C++ due to not default package manager/build system and too much divergence between platforms (e.g. unsigned long), not JS because their core build tooling changes to much (through it's getting better!), etc.)
> But for very small very well contained libraries I would argue that they sometime _are_ done
https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...
> As stated by McIlroy, and generally accepted throughout the Unix community, Unix programs have always been expected to follow the concept of DOTADIW, or "Do One Thing And Do It Well." There are limited sources for the acronym DOTADIW on the Internet, but it is discussed at length during the development and packaging of new operating systems, especially in the Linux community.
> As Steve Jobs observed, 8 one of the major differences between hardware and software is that for hardware the “dynamic range” (ratio between the best in class and average performance) is, at most, 2:1. But, the difference between the best software developer and an average software developer can be 50:1, or even 100:1, and putting great developers on a team with other great developers amplifies this effect. Today, in DoD and the industrial base that supports it, the people with the necessary skills exist, but instead of taking advantage of their skills we put them in environments where it is difficult for them to be effective. DoD does not take advantage of already existing military and civilian personnel expertise by offering pay bonuses, career paths that provide the ability to stay in their specialization, or access to early promotions. Skilled software engineers and the related specialties that are part of the overall software ecosystem need to be treated as a special force; the United States must harness their talent for the great benefits that it can provide.
As if we software people didn't have overinflated egos already...
This DoD report talks about 100x developers existing (and of course HNers imagine ourselves towards that better-than-average end of the scale).
The DoD here even uses the (lowercase) term "special force", which sounds like we're not nerd typists, but super-cool elite operators.
Ironically I suspect that wind l while the idea will feed many egos the reality would crush or frustrate those same people. If the DoD certifies someone as a 100x developer it would be after crafting a ruthlessly banal evaluation system that would likely lack the romantic attitude towards software many developers have in my experience. Especially if the criteria included some social skills component that determined part of a developers rating by how well they synergize with similarly rated peers.
I think the DoD has had many outstanding R&D successes, hiring or contracting elite engineers and scientists to do amazing things, which beat most of what we see from tech private sector B2C and B2B businesses.
That's one place you'll need some people who can find brilliant solutions or insights that most people simply cannot, no matter whether you throw 100x times the number of people at the problem.
But if they're having trouble attracting and retaining as many 1x-3x everyday software developers as they'd like, I agree, it'd be easy to imagine an attempt that fails miserably due to stereotypical bureaucracy.
Paying close to FAANG TC levels would be a start, since it would cover the monetary factor, and i think the prestige factor would quickly follow. (I bet few people think adtech, fintech, or yet-another-microblogging/messaging/profile-app are actually cool anymore -- only the money is what makes those companies prestigious.)
Maybe pay 80% of FAANG, so the FAANGs will still get the people who only want money, and you offer better non-monetary value to make up that 20%. (80% should be close enough they can afford to live in the same neighborhoods and send their kids to the same schools as FAANG, rather than be lower class that could upgrade just by grinding Leetcode for a few months.)
Then manage those units like they are very serious excellent product development organizations where everyone wants to work. And figure out how to do the culture better than all the companies that got too much into class thinking, and too much jumping through the hoops for interview and pursuit of promotions, rather than focus on the mission and team.
I feel Scrum has evolved into something contradicting the original Agile practices.
These were more of a template on how to develop, not a concrete recipe. Scrum then established a process where there was need for a family of processes, continuously evolving, ever changing.
The very fact that "you absolutely have to have a specific ceremony (i.e.: Retro when no topics)" is a red flag which imho puts the cart before the horse.
Why is there a "Scrum Master" anyway, if individuals are encouraged to interact and own the process? Suppose an individual wants to change the process, and reaches consensus within the team/community. Why would a Scrum Master / Manager even be able to veto any decision?
Why is there a Sprint when SW development is actually a marathon? Uncle Bob talks about this in several of his Clean Methodology books.
I don't think that Agile is dead [1], although it has been taken over by people eager to sell you a *cough* certification (Scrum or any other). It's not dead because its genius is that it's constantly evolving, and you can always interpret it differently since it's only a template. Scrum, on the other hand.. well..
For the future, I look forward to not putting an equal sign between Scrum and Agile, even though apparently Scrum is how you "do" Agile, according to the industry.
[1] https://www.simplethread.com/agile-at-20-the-failed-rebellio...
The problem isn't that DoD requires waterfall for software. The problem is the Planning Programming Budgeting Execution cycle. If you solve a problem, you have at least an 18 month hole in the schedule. This drives PMs to waterfall so the can project funding out over the POM hole. It's profoundly broken and kept that way by Congress critters in the pocket of the prime integrators.
A question I have: do people working on DoD software ever get to use it themselves? I assume many of them don't (if they're working on, say, a weapons system).
This situation often (IMHO) makes it much harder to deliver high-quality software. No amount of surface-level UI testing replaces the knowledge you get from actually using a system yourself.
The 10 recommendations found on the "cheat sheet" (page xv):
Line of Effort A (Congress and OSD):
Refactor statutes, regulations, and processes for software
Line of Effort B (OSD and Services):Create and maintain cross-program/cross-Service digital infrastructure
Line of Effort C (Services and OSD):Create new paths for digital talent (especially internal talent)
Line of Effort D (DoD and industry):Change the practice of how software is procured and developed
I didn't read through the whole thing, but I did skip to the section of the ten recommendations and each one makes a whole heap of sense to me.
Anyone know if Congress has done any of these things in the last three years, since this report has come out?
The attitude toward the development process expressed here is not uniformly accepted in all programming subcultures. There is a major cultural divide between embedded and server/desktop programming. My example is very old, but I had occasion to discuss software development with embedded systems engineers from Air Force projects in the pre-Ada days. They were all vehemently against the idea of software maintenance. They felt that their job was to deliver a "correct" implementation of a fully specified system.
That model makes sense for aircraft, in my opinion. The software isn’t a standalone product that can be independently continuously improved. It’s tightly integrated with safety critical physical systems. You need to do the upfront work of specifying how the software and hardware interact and do the complex systems safety analysis. Any change to the software specification after design can have unintended effects on the overall behavior of the aircraft.
> In a year replete with reports on artificial intelligence, quantum computing, and blockchain, may seem mundane to write about software, but software is the foundation of all things digital, and the Department does it exceedingly poorly.
I take issue with the title. Software is constantly "done" and then new requirements arrive, causing a new version of the completed software to be built. A suspension bridge is considered "done" even if it's getting continuously repainted.
You are thinking in terms of software in the non-government world. Contractors try really hard never to finish software so they can keep extending their contracts. This document is about the military world, and if you don't have experience in it, it can be hard to interpret!
military world only?
Look at how the TV series industry is hard trying with cliffhangers or story telling arcs without a closure...
This issue runs much deeper:
It is how the economy operates: you need a roof to protect you, every day, you need to eat, every day, until you die in decades. Many economic activities, if done honestly and fairly, cannot justify a permanent income.
Nearly all software has dependencies that need to be updated over time. How does that work against the bride analogy?
What is meant by done? As part of system it co-evolves with it and due with it. It dies with the system it sort of “parasite” on or it is the key part of the system or it is the system. Still evolve until death.
It's like reading a report by Darth Vader about the "national defense" of the galactic empire, explaining what's necessary for crushing "our adversaries" the rebels.
Chill down my spine.