Reminds me of how different DARPA was in the early years, compared to now.
If you're interested in how the modern Agency functions, I'd suggest reading The DARPA Model for Transformative Technologies.
It describes some of the typical trends in our time: contracts too often go to entrenched players and primes, rather than startups as primes due to technical merit rather than past performance and relationships. DARPA personnel reaching out from anonymous mailboxes post-RFI submission, asking if your team has TS/Sci clearance. (No answer if it's not a complete 100% yes.) And landlordism: extracting value from an existing system rather than seeding many new technologies.
However, there is still nothing like DARPA, despite its many problems. DIU is more for near-term innovation, and had its budget cut 20% just now. AFRL, AFWERX, Army SBIR/STTR, and so on, are all critical but not built to prevent technological surprise. Thank your lucky stars we have DARPA still up and running after so many decades.
To me the biggest sign that this is from a different era is seeing a Director personally generate something of value! It seems like leadership are all figureheads or apparatchiks or MBAs these days, usually subtracting from the mission rather than adding.
The cult of professional administration is the single most destructive thing in America today, in my opinion.
Not GP, but for me the phrase is referring to the idea that everything needs to be professionally administered like an enterprise business (e.g. government, colleges, etc.) with organizational structures that reflect that. It typically values process and procedure over curiosity and innovation.
This is a cultural plague in pretty much all activity these days. Even in the age of memes, influencers, and user-generated content, everything has been professionalized,
Professional sports has overwhelmed real amateur leagues. The Music Industry has displaced a lot of just getting together and singing badly. Wedding planners, summer gap year planners, even professional Christmas tree decorators. There are few roadside restaurants that don’t basically reheat something delivered frozen by CISCO. Random roadside motels and amateurish amusement park attractions have vanished in the face of chains. Even tinkering happens at a high level.
Sure, all the non-professional things still exist, but are a miniscule fraction of how they used to be.
I was more talking about the cultists themselves, and what sort of person becomes a professional administrator (hint: very few have ever actually directly built or accomplished anything more complicated than assembling a piece of IKEA furniture, and most of them even have to subcontract that). The collective noun for cultists is cult, hence my words.
Their collective delusion is that a business or other effort can be administered without actually understanding what it is and does. This is lethal to the business.
That said, an actually good, down-to-earth professional administrator is worth their weight in platinum. These types are very, very rare.
I have worked with ERC and many other funders, as well as in DARPA-funded projects. Talked to principal investigators, attended panel reviews, seen how projects are evaluated, etc.
My impression is that, despite some problems, DARPA is outstanding compared to what we have across the pond. Obviously, that is not a reason to ignore issues or try to fix them.
Yes -- I hear encouraging things about ARIA. It hasn't been stood up yet, but Peter Highnam, former DARPA deputy director, was supposed to be the first ARIA director, if I recall correctly. I met Peter very briefly, years ago. Good guy.
I think Grand/Urban Challenge was the last great project and success. The openness for participation was unprecedented resulting in lot of different startups taking part (from a "serious" point of view for example that Lewandowski autonomous bike was in the GC 2004 basically an utter laughing stock of a failure unimaginable in any "serious" setting, yet it was there too, and happened to be one of the key starting points of the modern autonomous industry). I think that success kind of scared the "serious" system so that even for Urban Challenge they already had a parallel "crony" track, and seems there haven't been such an open project since then.
Wrt. original post - interesting how the catechism is different from the today's dominating approach of "we don't really know what we want, just have a very vague set of desires, lets try to achieve that small goal for that sprint which hopefully will move us in some desired direction, and we'll see after the sprint where it would lead us, and where we can make the next step from there."
Honestly, this seems like a great way to articulate one's goals with any complex project. As a teacher, I could imagine using this framework to describe a new curriculum.
> What are you trying to do? Articulate your objectives using absolutely no jargon.
I'm trying to improve reading skills for all students.
> How is it done today, and what are the limits of current practice?
Currently, students are grouped by ability (”reading level") for reading instruction. This means that students who are considered below grade level are rarely exposed texts at grade-level complexity.
> What is new in your approach and why do you think it will be successful?
Instead of grouping students by ability, I will create mixed-ability student groups to read grade-level texts with all students. I will provide differentiated support for students who cannot yet read selected texts independently. This approach is supported by Young's (2022) research, which detailed greater reading level gains for emerging readers.
> Who cares? If you are successful, what difference will it make?
Students who are considered below grade level in reading will show increased gains compared to the previous approach.
> What are the risks?
This approach requires more reading instruction expertise from teachers. Teachers who are unable to adequately support below-grade-level readers risk further delaying these readers' progress. Because this is a change to widespread and currently accepted instructional practices, it is likely that many teachers and parents will resist changes to the program, either explicitly or by falling back to previous practices.
> How much will it cost?
The only significant cost is in teacher professional development, which I estimate to be $X for the first academic year and $Y in subsequent years.
> How long will it take?
The new approach will be rolled out within the first 2 months of the school year, with ongoing professional development throughout the year.
> What are the mid-term and final “exams” to check for success?
We will continue to rely on the same reading assessments we have been using. We will also conduct qualitative evaluations of teachers' and students' attitudes toward the program in November, February, and June.
I’ve been using the catechism in everything I write throughout my PhD. It’s real a catechism for academia, it’s all about answering these questions in one way or another.
I just start with it and start adding bullet points to answer these questions, then I put more bullet points and expand in them, and so on until I have enough material. Then I map it to each section of w/e document I am writing.
It’s been one of the most useful things in I’ve ever learnt about. Came to learn about it from an IEEE webinar by a professor of power systems (Siddharth Suryanarayanan) that I stumbled upon via vEvents when COVID-19 lockdowns started. I presented a copy of it to my lab in my PhD, helped my lab-mates clarify things to their supervisor and write. It was super effective.
This and the W6H are what I tell any new developer to include when briefing their efforts to clients or management.
The W6H overlaps the Heilmeier Catechism some:
- Who: Who will perform the work?
- What: What work will be performed?
- Where: Where will the work be performed? Is the env. suitable or are purchases needed? If purchases, what?
- Why: Why are you/we the people to perform this work? Why does this work need to be performed? Why are we not using an off-the-shelf solution (this is where you can present an Analysis of Alternatives and/or Gap Analysis)?
- When: When will the work start? When is it expected to end? When is it needed and what is it needed by?
- Which: Which customer are you performing the work for?
- How: What are your planned methods, at a highlevel, and reasoning behind those choices?
The original source for the above is Aristotle, but thanks to Wikipedia [1] I've always thought the way I use them to more match Hermagoras of Temnos, as quoted in pseudo-Augustine's De Rhetorica[2]:
Quis, quid, quando, ubi, cur, quem ad modum, quibus adminiculis.[3]
(Who, what, when, where, why, in what way, by what means)
[2] Cecil W. Wooten, George Alexander Kennedy, eds., The orator in action and theory in Greece and Rome, 2001. ISBN 90-04-12213-3, p. 36.
[3] Robertson, D.W. Jr (1946). "A Note on the Classical Origin of "Circumstances" in the Medieval Confessional". Studies in Philology. 43 (1): 6–14. JSTOR 4172741.
Plus additional see also, the recent Hacker News thread on How To improve one's Technical Writing, the link to which I'll ninja-edit-insert here in a second:
In my opinion, historically, the decisions which led to IBM's Zachman Framework only utilizing W5H(=6 questions) instead of W6H (=7 questions) really screwed us all over by whole freightloads more than almost anyone realizes as it has horribly miscalibrated more societal systems(') environments with ground truth biasing so extensively that it has started to slip into recursive, some sort of eldritch generalization of xkcd's Citogenesis cycle.
I hadn’t seen this before and I’m 100% going to steal it and use it for my next product/feature/business case pitch. At least as a start to prove value to myself.
I’m fully aware of the utility of jargon in writing a business case, it lets me skip over the difficult, the warts, and the questionable. I wonder if there’s a correlation between failed projects and how jargon heavy the initiation is?
I also really like thinking about milestones as exam points. Calling them exams for success more directly gets at the point- to check in and course correct.
> I’m fully aware of the utility of jargon in writing a business case, it lets me skip over the difficult, the warts, and the questionable. I wonder if there’s a correlation between failed projects and how jargon heavy the initiation is?
I'll say this, based on my experience working at a particularly dysfunctional organization: in these kinds of environments, skipping over the most obvious questions seems to be the norm. Most of the time, when I was exposed to a new project, the explanation gave almost minimal explanation about fundamentals such as:
1. What is the time scale of the project? (e.g. weeks? (probably not), months? (that would be a moonshot), about a year? (maybe...), many years (yikes!))
2. What phase is the project in? (e.g. conceptualization, technical design, implementation, etc)
3. What specific teams and people are working on the project? Who is in charge? What kind of leadership will they offer? (e.g. selling the project up the ladder, project-level, technical, etc) ...
Also, the following questions were usually in play:
4. Where can I go read about the project details?
5. Why does it seem that each person who hears about the project has a very different idea of what it is supposed to do?
6. Wasn't there a previous version of this project? What was it called again? Did it ever launch? Why didn't it ever go anywhere?
I learned about these from my PhD advisor and I’ve used them for every research project ever since. It’s been very useful and I refer to it every month or so.
It's like a scientific method except with an added minimal step of abstraction. It's like a rule for technical writing except the writing is a kind of proof. It's a clarifier, or a bullshit-filter. This kind of precise language-use really thrills me.
Reminds me of how different DARPA was in the early years, compared to now.
If you're interested in how the modern Agency functions, I'd suggest reading The DARPA Model for Transformative Technologies.
It describes some of the typical trends in our time: contracts too often go to entrenched players and primes, rather than startups as primes due to technical merit rather than past performance and relationships. DARPA personnel reaching out from anonymous mailboxes post-RFI submission, asking if your team has TS/Sci clearance. (No answer if it's not a complete 100% yes.) And landlordism: extracting value from an existing system rather than seeding many new technologies.
However, there is still nothing like DARPA, despite its many problems. DIU is more for near-term innovation, and had its budget cut 20% just now. AFRL, AFWERX, Army SBIR/STTR, and so on, are all critical but not built to prevent technological surprise. Thank your lucky stars we have DARPA still up and running after so many decades.
To me the biggest sign that this is from a different era is seeing a Director personally generate something of value! It seems like leadership are all figureheads or apparatchiks or MBAs these days, usually subtracting from the mission rather than adding.
The cult of professional administration is the single most destructive thing in America today, in my opinion.
What is the cult of professional administration?
Not GP, but for me the phrase is referring to the idea that everything needs to be professionally administered like an enterprise business (e.g. government, colleges, etc.) with organizational structures that reflect that. It typically values process and procedure over curiosity and innovation.
This is a cultural plague in pretty much all activity these days. Even in the age of memes, influencers, and user-generated content, everything has been professionalized,
Professional sports has overwhelmed real amateur leagues. The Music Industry has displaced a lot of just getting together and singing badly. Wedding planners, summer gap year planners, even professional Christmas tree decorators. There are few roadside restaurants that don’t basically reheat something delivered frozen by CISCO. Random roadside motels and amateurish amusement park attractions have vanished in the face of chains. Even tinkering happens at a high level.
Sure, all the non-professional things still exist, but are a miniscule fraction of how they used to be.
> delivered frozen by CISCO.
I think you mean Sysco. CISCO sell routers, Sysco sells food
Yes, brain fart. Thanks
I was more talking about the cultists themselves, and what sort of person becomes a professional administrator (hint: very few have ever actually directly built or accomplished anything more complicated than assembling a piece of IKEA furniture, and most of them even have to subcontract that). The collective noun for cultists is cult, hence my words.
Their collective delusion is that a business or other effort can be administered without actually understanding what it is and does. This is lethal to the business.
That said, an actually good, down-to-earth professional administrator is worth their weight in platinum. These types are very, very rare.
I have worked with ERC and many other funders, as well as in DARPA-funded projects. Talked to principal investigators, attended panel reviews, seen how projects are evaluated, etc.
My impression is that, despite some problems, DARPA is outstanding compared to what we have across the pond. Obviously, that is not a reason to ignore issues or try to fix them.
In fact, new initiatives are often trying to mimic DARPA. For example, Wellcome LEAP: https://wellcomeleap.org/leap-innovation-model/
Yes -- I hear encouraging things about ARIA. It hasn't been stood up yet, but Peter Highnam, former DARPA deputy director, was supposed to be the first ARIA director, if I recall correctly. I met Peter very briefly, years ago. Good guy.
I think Grand/Urban Challenge was the last great project and success. The openness for participation was unprecedented resulting in lot of different startups taking part (from a "serious" point of view for example that Lewandowski autonomous bike was in the GC 2004 basically an utter laughing stock of a failure unimaginable in any "serious" setting, yet it was there too, and happened to be one of the key starting points of the modern autonomous industry). I think that success kind of scared the "serious" system so that even for Urban Challenge they already had a parallel "crony" track, and seems there haven't been such an open project since then.
Wrt. original post - interesting how the catechism is different from the today's dominating approach of "we don't really know what we want, just have a very vague set of desires, lets try to achieve that small goal for that sprint which hopefully will move us in some desired direction, and we'll see after the sprint where it would lead us, and where we can make the next step from there."
DARPA PPAML (probabilistic programming for ML) was pretty good.
It helped Stan reach maturity, and also led to things like Pyro.
But we need to wait a bit more to see where this goes. The program only finished 5 years ago.
Honestly, this seems like a great way to articulate one's goals with any complex project. As a teacher, I could imagine using this framework to describe a new curriculum.
> What are you trying to do? Articulate your objectives using absolutely no jargon.
I'm trying to improve reading skills for all students.
> How is it done today, and what are the limits of current practice?
Currently, students are grouped by ability (”reading level") for reading instruction. This means that students who are considered below grade level are rarely exposed texts at grade-level complexity.
> What is new in your approach and why do you think it will be successful?
Instead of grouping students by ability, I will create mixed-ability student groups to read grade-level texts with all students. I will provide differentiated support for students who cannot yet read selected texts independently. This approach is supported by Young's (2022) research, which detailed greater reading level gains for emerging readers.
> Who cares? If you are successful, what difference will it make?
Students who are considered below grade level in reading will show increased gains compared to the previous approach.
> What are the risks?
This approach requires more reading instruction expertise from teachers. Teachers who are unable to adequately support below-grade-level readers risk further delaying these readers' progress. Because this is a change to widespread and currently accepted instructional practices, it is likely that many teachers and parents will resist changes to the program, either explicitly or by falling back to previous practices.
> How much will it cost?
The only significant cost is in teacher professional development, which I estimate to be $X for the first academic year and $Y in subsequent years.
> How long will it take?
The new approach will be rolled out within the first 2 months of the school year, with ongoing professional development throughout the year.
> What are the mid-term and final “exams” to check for success?
We will continue to rely on the same reading assessments we have been using. We will also conduct qualitative evaluations of teachers' and students' attitudes toward the program in November, February, and June.
It also sounds like good questions to ask students who come to you with ideas for term projects, if your school does those.
I’ve been using the catechism in everything I write throughout my PhD. It’s real a catechism for academia, it’s all about answering these questions in one way or another.
I just start with it and start adding bullet points to answer these questions, then I put more bullet points and expand in them, and so on until I have enough material. Then I map it to each section of w/e document I am writing.
It’s been one of the most useful things in I’ve ever learnt about. Came to learn about it from an IEEE webinar by a professor of power systems (Siddharth Suryanarayanan) that I stumbled upon via vEvents when COVID-19 lockdowns started. I presented a copy of it to my lab in my PhD, helped my lab-mates clarify things to their supervisor and write. It was super effective.
Olin Shivers' dissertation advice is also very short and pointful:
https://www.ccs.neu.edu/home/shivers/diss-advice.html
This and the W6H are what I tell any new developer to include when briefing their efforts to clients or management.
The W6H overlaps the Heilmeier Catechism some:
The original source for the above is Aristotle, but thanks to Wikipedia [1] I've always thought the way I use them to more match Hermagoras of Temnos, as quoted in pseudo-Augustine's De Rhetorica[2]:Quis, quid, quando, ubi, cur, quem ad modum, quibus adminiculis.[3] (Who, what, when, where, why, in what way, by what means)
[1] https://en.wikipedia.org/wiki/Five_Ws
[2] Cecil W. Wooten, George Alexander Kennedy, eds., The orator in action and theory in Greece and Rome, 2001. ISBN 90-04-12213-3, p. 36.
[3] Robertson, D.W. Jr (1946). "A Note on the Classical Origin of "Circumstances" in the Medieval Confessional". Studies in Philology. 43 (1): 6–14. JSTOR 4172741.
Here's one more highly relevant citation into which I eventually stumbled, which covers additional details about the W6H:
https://www.academia.edu/29682686/The_Roots_of_News_in_Journ... Stamenković, Gordana — "The Roots of News in Journalism.", Unpublished Manuscript, 2016.
Plus additional see also, the recent Hacker News thread on How To improve one's Technical Writing, the link to which I'll ninja-edit-insert here in a second:
https://news.ycombinator.com/item?id=31859040#31877184
In my opinion, historically, the decisions which led to IBM's Zachman Framework only utilizing W5H(=6 questions) instead of W6H (=7 questions) really screwed us all over by whole freightloads more than almost anyone realizes as it has horribly miscalibrated more societal systems(') environments with ground truth biasing so extensively that it has started to slip into recursive, some sort of eldritch generalization of xkcd's Citogenesis cycle.
Related:
The Heilmeier Catechism (1977) - https://news.ycombinator.com/item?id=17272801 - June 2018 (19 comments)
I hadn’t seen this before and I’m 100% going to steal it and use it for my next product/feature/business case pitch. At least as a start to prove value to myself.
I’m fully aware of the utility of jargon in writing a business case, it lets me skip over the difficult, the warts, and the questionable. I wonder if there’s a correlation between failed projects and how jargon heavy the initiation is?
I also really like thinking about milestones as exam points. Calling them exams for success more directly gets at the point- to check in and course correct.
> I’m fully aware of the utility of jargon in writing a business case, it lets me skip over the difficult, the warts, and the questionable. I wonder if there’s a correlation between failed projects and how jargon heavy the initiation is?
I'll say this, based on my experience working at a particularly dysfunctional organization: in these kinds of environments, skipping over the most obvious questions seems to be the norm. Most of the time, when I was exposed to a new project, the explanation gave almost minimal explanation about fundamentals such as:
1. What is the time scale of the project? (e.g. weeks? (probably not), months? (that would be a moonshot), about a year? (maybe...), many years (yikes!))
2. What phase is the project in? (e.g. conceptualization, technical design, implementation, etc)
3. What specific teams and people are working on the project? Who is in charge? What kind of leadership will they offer? (e.g. selling the project up the ladder, project-level, technical, etc) ...
Also, the following questions were usually in play:
4. Where can I go read about the project details?
5. Why does it seem that each person who hears about the project has a very different idea of what it is supposed to do?
6. Wasn't there a previous version of this project? What was it called again? Did it ever launch? Why didn't it ever go anywhere?
I learned about these from my PhD advisor and I’ve used them for every research project ever since. It’s been very useful and I refer to it every month or so.
It's like a scientific method except with an added minimal step of abstraction. It's like a rule for technical writing except the writing is a kind of proof. It's a clarifier, or a bullshit-filter. This kind of precise language-use really thrills me.
We use this internally at mobilecoin as the base framework for new PRDs. Very clarifying.
> What are you trying to do? Articulate your objectives using absolutely no jargon.
I think this is very important, but I have seldom seen it in action. I tend to run into jargonauts, everywhere.
I hereby promise to post this every time: https://www.youtube.com/watch?v=roTdADxbClI