The federal government hires a team of young, talented, motivated engineers and managers, puts them in charge of failed software projects, gives them the resources and authority they require to turns things around, and -- surprise! -- it turns out they do a GREAT job.
In hindsight, this shouldn't be too surprising. What might be surprising is that the same logic should apply to ALL government functions, not just software development. Who says government projects and agencies have to be poorly run? Who says we can't devise better systems and/or mechanisms (e.g., market-based mechanisms) for attracting and retaining great people in government?
Disclosure: I'm an engineer at USDS and these are my own opinions.
So in my admittedly short time in the government [0], I've witnessed how all of these problems are due to good intentions. That's what makes this all really tough because everything you think is bonkers actually has a reason.
The 1400 page travel regulations is a result of trying to prevent fraud - every single issue that comes up results in a new rule.
The fact that it takes some projects years to deploy is that we would like to plan and make sure that every resource is well-spent, that it's in a number of languages and accessible to the blind.
It makes it hard for everyone - I've met lots of smart talented civil servants and government contractors who want to do things differently but have their hands tied behind their back.
[0] 2 years feels like forever to me but flash in the pan to many of the dedicated civil servants I've met.
Disclosure: I'm active duty Navy and these are my own opinions.
In 22 years, I have never been so hopeful for meaningful improvement in my work life as I am now. Having met a few folks I am all too familiar with DTS and the JFTR (the 1400 pages in the article(1)). I think that's a great choice to start with: like Google going after the mundane problems of every person's life. This will make a difference. I am on travel now and was on the phone and DTS (simultaneously) for an hour today. And for anyone who tries to apologize for the 1400 pages, please don't. I have cut instructions from 238 pages to less than 30. I would argue the major problem is not that people are trying to solve every edge case. The major problem is that people are only in a job for a short period of time, come in, and while they may try to solve the edge cases they encounter, they often do that by trying to simplify things by inserting a new abstraction and taking ownership of that abstraction. So the layers of abstraction accrete like sediment. And as long as there's no direct logic conflicts, they can promote away from the problem.
I will gladly buy any USDS, 18F, or DDS hacker in San Diego a beer. Keep up the good work.
> every single issue that comes up results in a new rule.
This sentence is the simplest explanation for government (and bureaucratic) incompetence.
Think about writing software. Is the optimal solution to every single bug to write more code to deal with that specific situation? Of course not. In many cases, sorting out the underlying cause and fixing that (which may involve new code, rewriting old code, or even deleting outdated code) is the correct approach (assuming that the optimal solution is the desired outcome, there are of course cases where speed of getting something out that works trumps this, but government regulations only take effect once annually in most cases anyway, so they don't have a speed excuse).
Simply writing a new rule to deal with every scenario is an approach that inevitably leads here.
You demonstrate intellectual fallacies common amongst software developers:
1. Believing that your experience in one particular field makes you qualified to make pronouncements regarding others about which you know nothing.
2. Believing that problems in other fields are inherently simple to understand and solve, and that the reason they aren't must therefore be due to the malice or incompetence of people working in those fields.
3. Believing that software and software development processes are a universal model that can be applied to any other problem via trite and reductive analogy.
With these considerations taken we can apply the same problems to people in industrial scale manufacturing that have typically managed many organizations that are now in charge of software projects. Then comes no real effective incentive structure with how most federal contracts are written despite billions spent on lawyers to protect the government and to get the most for the government's dollars supposedly. I've seen far too many projects with clearly the most talented and well-managed contractors getting tossed for actually finishing their deliverables while those that didn't get through 40%+ got renewed because they were just too critical to the political success of the greater project. And they'll continue to cite the project as past performance and renewal as an indicator of competence.
The fact there's so much opposition to USDS and 18F that's greater than any other group ever is the best indicator that the fat, bloated Beltway bandits are worried about their easy road to retirement. This isn't to say everyone's lazy - far from it. But government contracting has been largely insulated from the realities of most commercial enterprises through the politicized veil of "protecting veterans" and "defending the country" and for every legitimate, honest worker there's at least two that just want a cushy 9-5 for 35 years.
while i frequently agree with you, i disagree here.
both code and travel regulations are lists of rules written in a terse language, and both are subject to bloat all of the time, abd parts become deprecated as times and priorities change.
these are very comparable things, e cept that code can change quickly at a low cost, while regulations are costly to change.
Because of the cheapness, software folks have put a lot of effort and study into optimizing how you make rule changes, which makes it very applicable to some other problem spaces
> ...except that code can change quickly at a low cost, while regulations are costly to change.
And you don't think that's a salient difference? Particularly in light of the adversarial nature of politics, which was one of your comment's parent's points, and which is a major contributing factor to such changes being so costly?
Yet the information density of the parent comment is so much higher. You have 3 statements that can be reduced to "Things that work in programming don't necessarily work in other fields." No kidding? Then what's your solution for operational efficiency in government?
Some of us believe detail and nuance add power to argument, rather than rejecting them in favour throwing around basic, unsupported claims.
And your reduction is not even correct. Assuming that problems in other fields are inherently easy to solve has nothing to do with the applicability of software engineering techniques. Assuming those working in other fields are incompetent or malicious has nothing to do with the applicability of software engineering techniques. You say my post was lacking in information density, yet you apparently weren't even able to grasp the arguments I did make. So maybe I needed more explanation, not less?
And are you implying that unless I come up with a solution for efficient government, it somehow renders my — entirely unrelated — argument invalid? That's nothing more than a lame attempt at argumental misdirection. But hey, I'll bite: My solution for operation efficiency in government is for everyone to think and act in the complete opposite manner to you. Is that reductive enough for you?
> And are you implying that unless I come up with a solution for efficient government, it somehow renders my — entirely unrelated — argument invalid?
I'm implying that you're long-winded and it weakens your argument. If I reduced your response, I'd reduce it to, "You hurt my feelings and I'm angry about that." Fair enough, but the rest is so much filler.
Consider the software that NASA writes and how it's written. Rigorously specified, reviewed, and tested by some of the best engineers in the world to the point that almost bug-free code is produced at the expense of a much slower rate of development. Which is the best you can do with billions of dollars and human lives on the line for certain projects.
Now look at most government software infrastructure: frequently mercurial and ambiguous software specifications interacting and/or based on flawed laws and regulations filled with logical contradictions written by Congressmen and lobbyists with perverse incentives. And you have to justify every cent or risk the accusation of wasting taxpayer dollars.
In the shuttle group's culture, there are no superstar programmers. The whole approach to developing software is intentionally designed not to rely on any particular person.
And the culture is equally intolerant of creativity, the individual coding flourishes and styles that are the signature of the all-night software world. "People ask, doesn't this process stifle creativity? You have to do exactly what the manual says, and you've got someone looking over your shoulder," says Keller. "The answer is, yes, the process does stifle creativity."
I think equating "best engineers" with "superstars" means you might be bringing your own associations to the topic. (Not unfairly, that's a standard association in the Valley, but still.)
The few NASA engineers I've known have been superb as NASA employees. They weren't grand innovators solving problems on their own, but they were knowledgeable and intelligent. They had deep understanding of the tools they worked with, were rigorously careful and formal, and understood the problems and tradeoffs of their work far beyond any spec they were handed.
To me, that counts as being one of the best engineers in the world. These are people who know what they need to do, why they need to do it, and how they can best accomplish it. In the case of NASA, that generally means doing something radically different than you would at a tech startup, but these people are still brining enormous ability and great care to their work.
The same logic applies to all those regulations that seem bonkers.
If you think of all those regulations as a type of source code (which dictates what government employees and citizens can and cannot do, when, and under what conditions), it's clear that a lot of regulatory code needs major refactoring.
To use your example with travel regulations, those 1400 pages designed to prevent fraud likely consist primarily of thousands upon thousands of assertions and if-then statements. I wonder if it would be possible to reduce them to, say, a few dozen pages -- by refactoring all that 'regulatory code' to use different, higher-level abstractions.
My guess is that most of those 1400 pages consist of the corner cases - things that will 95% of the time never pop up, but when they do will wreak havoc unless properly dealt with. Granted, I'm not sure travel fraud can wreak that much havoc, but I'm sure someone somewhere gets annoyed over it.
My (admittedly limited) experience in coding/engineering has taught me that it's unwise to look at technical problems and people problems in the same light; the former can be solved much easier than the latter. The trick is to figure out where you can solve the former to avoid the latter!
Not in government, but I understand the biggest problem with travel to be the expense account, which has a history of being abused to disgusting proportions.
Frankly, my take is that this is what happens when you try to beat human intent with formal rulings. Explicitly banning every possible form of travel fraud is almost unimaginable - certainly it can't be done without banning a huge amount of legitimate travel also. Rigorous safety around people problems is nigh-impossible, which is why most safe software systems take the approach of "do it our way or go to hell".
At a certain point you can only solve the people problems with oversight and good intentions. You could get one random employee to certify any given travel plan or reciept as "not obviously fraudulent" and recreate the benefit of ~700 pages of regulations, just by showing the thing to someone who doesn't benefit from fraud.
But of course, incremental change produces these kind of awful local minima. If you are punished for fraud, aren't punished for overhead, and can't change the whole system, what else would you do? You ban one known misbehavior, go on with your day, and everything gets a little bit worse.
"The 1400 page travel regulations is a result of trying to prevent fraud - every single issue that comes up results in a new rule."
This seems like a serious inability to understand that no process designed to prevent future things you can't forsee is 100% effective (by definition). At some point, you have to declare "good enough", and live with it until the error rate becomes unacceptable overall again, then modify it.
IE it's likely 50 pages of those regulations gave them a 99.9%+ rate of avoiding fraud. They then added 1350 pages to get to probably 99.99%
This is unlikely to be worth it.
(and yes, before someone points it out, i'm likely being generous with the numbers)
Some of it comes down to agencies needing to protect themselves from congressional witch-hunts. Consider a congressman or political party that has ideological objections to an agency even existing and wishes to neuter or eliminate it. A stellar way of achieving this is by making the target seem wasteful and fraudulent [1]. If there is actual fraud, no matter how small or how much of a corner case, this task becomes even easier.
1. A good example of this is how Republicans periodically attempt to defund science agencies by mocking research projects that sound frivolous.
A Democrat, the late Senator William Proxmire of Wisconsin, was well known for mocking frivolous-sounding research projects. From the Wikipedia article[0]:
"In 1987, Stewart Brand accused Proxmire of recklessly attacking legitimate research for the crass purpose of furthering his own political career, with gross indifference as to whether his assertions were true or false as well as the long-term effects on American science and technology policy."
It's good to see examples from both sides. That being said, what the above mentions is Republican doctrine, as opposed to isolated cases of politicians(on either side) simply blindly furthering their political careers.
Unfortunately there are a lot of people who believe government shouldn't do anything unless it is fraud-proof. The narratives around welfare and food stamp abuse make headlines for exactly this reason :(.
I think their intentions are a little more nefarious. It's not that they want e.g. a fraud-free welfare system; they fundamentally disagree with the idea of welfare and so use fraud, whether it's a legitimate issue or not, as a basis for trying to stymie or dismantle the institution.
The problem is, once the government chooses to not close a known loophole, the number of people who exploit it may increase by orders of magnitude. Without a willingness to add the other 1350 pages, you may end up with something like 70% fraud prevention, not 99.9%.
What's needed is more refactoring. This would benefit from more capacity to try different sets of regulations in parallel.
This is a generally true statement about any process.
The solution to that is to enforce well enough that people don't think that's a good idea. I also did say you do have to refactor over time as compliance rate decreases.
Past that, i don't think we actually disagree :)
If you have a speed limit sign, and it says "speed limit, 50 mph, enforced by satellite observation", most people will probably ignore it. Those that don't and get caught, yeah, they go looking for excuses for why they ignored it to post-justify it.
Changing the regulation wording will not change this. You can make the sign much larger and say "speed limit 50 mph, even if you are really late for an appointment, etc" but honestly, it still will not help that. People ignore it because the enforcement mechanism makes them feel like it won't happen to them (and because it's not socially abhorrent, etc), not because of ignorance of the law
On the other hand, if you have a sign that says "speed limit 50mph, enforced by this guy, right here", and there is a smiling cop with a radar gun sitting next to the sign, enforcing it, most people will not ignore it. In fact, i'd bet you could write everything before "enforced by this guy" in small print people had to slow down to read, and most people would slow down and read it, because they believe the risk of enforcement is greater to them.
Will you get everyone to stop speeding there?
Nope.
Even if you add spike strips, laser beams, whatever, someone is going to do it, and in fact, enforcing harder sometimes increases the rate (depending how low the rate is) based on the thrill some people get. 100% compliance is just pretty much impossible, no matter what words you use.
Some organizations do startlingly well with good enforcement and a rule against circumventing the rules. Yes, that's subjective and messy, but it can actually work quite nicely.
Hell, it's basically what financial structuring laws are: a rule saying "no using loopholes if you find them". With that in place, it becomes surprisingly easy to address loopholes by punishing everyone who employs them.
A common approach is to set a fixed amount per day for expenses based on cost levels in the country in question, and be extremely strict with extras, coupled with approved supplier lists and price ranges for the actual travel.
It "rewards" those who are prudent with extra cash, and so it certainly won't be perfectly efficient, but in return it makes it harder for those who would otherwise try to abuse the system who often will go far overboard, because any extra expensive claims can be given a lot more attention (and often will require advance approval), and it drastically cuts down on paperwork.
But unlike security issues a single failure doesn't compromise 100% of the rest of the system. This is also why analogies between software/security/cryptography/privacy and the tangible world are so awkward.
Fraud prevention actually is a security issue. Not an Internet security issue, so mistakes aren't punished that quickly, but the analogy is still sound.
Someone buying a new watch with their expense account doesn't suddenly give them access to the whole treasury -- that's the difference between physical and digital realms I am trying to emphasize.
Most security breaches don't allow the malicious user to root the entire server farm either.
I just spent a week fixing permission validation done in JS on the browser. Users could have potentially allowed themselves to see parts of documents outside their role. This didn't give them access to our payroll system, credit card processor, or the backend infrastructure.
This is a big part of the answer. Congressional hearings and reporting often act like "fraud is fraud", but allowing 1% fraud to save 20% overhead is entirely reasonable.
Improper resource usage is a better metaphor than security failures for this topic.
People are pretty sensitive about government financial workers committing fraud, similar to how they are rather sensitive to government police committing murder.
Sadly, in neither case will you ever have 100% compliance.
Pretending it's achievable, and trying to achieve it, is IMHO, silly.
Remember the regulations do not prevent fraud, enforcement prevents fraud. There already exist plenty of things saying it's not okay, etc. Saying "and also, don't do that" is probably not actually necessary most of the time, in the same way saying "don't shoot people" is sufficient. Saying "and also don't shoot them while they are handcuffed" isn't necessary. Crappy post-justification does mean the regulation was written wrong, and changing the regulation to account for the post-justification will not actually improve the process most of the time.
I don't think we can take this much further without knowing what's actually in the regulations, but I imagine they consist more of "officer's dash cam will be run 24/7 and backed up in triplicate", "officer will learn proper gun handling techniques X, Y, & Z", etc rather than "don't shoot people", "don't shoot handcuffed people", "don't shoot clowns", "don't shoot children".
Or, in the fraud case, "books will be audited at frequency X", "Y behavior makes it too easy to hide fraud and is not allowed". Rather than "fraud is illegal on Monday", "fraud is also illegal on Tuesday", "fraud is even illegal on holidays"...
Of course we can never achieve 100% with more regulation, but we make it more of a priority to make abuse harder to get away with than elsewhere, presumably increasing overhead in exchange for lowering abuse (yes, this is probably not a strictly linear curve)
There are some sensible regulations there, like having someone approve travel requests, but there are also a lot of very narrow restrictions obviously added by someone who wanted to prevent Fraud X, but lacked the authority to change what was already written. The result is that you get more overhead with depressingly little payoff.
In principle you're right about the trade-off, but that's only the case when rule-writers have the authority to sensibly restructure what already exists.
This is a good summary. At a certain point, you honestly can just have a rule against stupid or malicious behavior. The trick is to enforce it carefully and sensibly, rather than to pursue comprehensive objective rules.
Anyone who's played rules-lawyering games like Nomic will be aware that banning all misbehavior explicitly is impossible. You're basically limited to whitelisting approved behaviors, or implementing a general rule against malfeasance. Unless the consequences of misbehavior are enormous, the second option tends to be more efficient.
I don't think that necessarily has to be the case. The public conversation could conceivably shift to a cost/benefit analysis of varying levels of enforcement vs. fraud, if only the media would cooperate.
This is definitely the case, because we already see different analyses for different topics.
When it comes to NSF, people worry about overhead and waste. When it's welfare or food stamps, people worry about fraud instead. Some of this is moral - people care about the 'undeserving poor' more than 'undeserving scientists' - because we tend to hate abuse of charity. But it clearly shows that there are different categories of concern, and that the public is capable of examining both topics.
This is exactly the problem. When you have a system that completely ignores inefficiency/overhead but goes berserk over fraud, you get totally absurd incentives. Those 1350 pages probably kept some managers from getting fired, but realistically have been a serious waste of time and money.
At a certain point, you either accept a low level of fraud or just make a rule saying "don't do bad, wasteful stuff." Then you fire anyone who breaks that rule and let things work themselves out. (This has other problems, but they can be addressed.)
Most of bureaucratic stupidity is ultimately moral hazard. Someone pays for one failure case but not another, so they spend absurd amounts minimizing what they're responsible for.
I want to work there so bad. USDS or 18F. But I can't get anyone to call me back, even with twenty years working in (and running) startups. Dunno what that's about.
Hi Fapjacks. I'm on the talent team at 18F. Feel free to email me directly at amanda.schonfeld@gsa.gov. We don't have a direct phone number for folks to call, but I will definitely email you back. :)
Very informative comment, but then, did that change once the usds were in charge ? Because my intuition is that taking over an openly failed project makes it easy for the new team to tell everyone to try and keep things simple.
Then wouldn't the succes not be a matter of technical abilities or process, but rather goodwill by the client side to remain reasonable ?
"gives them the resources and authority they require to turns things around"
They probably could have given the same resources and authority to the existing staff and they might have turned things around too. It really annoys me that in so many cases the existing guys know what's wrong but are not allowed by management to do anything about it. Then management hires new people, gives them authority, and suddenly these guys are heroes and the existing people look like idiots.
There seems to be this tendency to think that existent workers are all idiots and the only way to do better is to hire new people (preferably young and from ivy league schools). In reality it's a leadership problem because they have set up dysfunctional environments that don't allow people to perform.
Disclosure: I'm an engineer at USDS and these are my own opinions.
We are very proud to be working along side the existing agency and contracting staff. You are right that many times, they do know how to fix it and need some help from us getting management to let them do it. Some times they are skeptical, sometimes there is friction - that's all completely understandable.
Ultimately, my time in government has opened my eyes to the talented folks inside government right now and I'm proud to be working along side them. I hope you didn't get a poor impression of us and the work that is happening.
I didn't mean to offend you at all. USDS is great. I just see this dismissive attitude of existing staff by management. But in reality it's management that messed things up, not staff.
Thanks. I appreciate that. I think you actually have a keen eye for the issues.
Since my time in government, I've become more sympathetic to agency employees, contractors, management - pretty much everyone. The entire environment is really challenging to do the right thing.
As an executive, the typical tools you have at your disposal, budget and HR, are circumscribed by congressional appropriations and federal hiring guidelines. Your ability to promote and demote and reorganize are limited by laws and unions and you definitely can't grant stock options. The metrics that you typically use to measure success are much fuzzier because there isn't a PNL but it's really easy for the press to find a single case where your org does a poor job.
I still maintain hope that good things can be done - I have seen it - and good management makes a huge difference.
Disclosure: I'm an engineer at USDS and these are my own opinions.
If you're comparing to the paper bureaucracies of yore, our bureaucracy isn't even that bad. I mean, look at the size of the country and the number of functions, they ran that on PAPER until the 70s-80s?
The problem with the software transition is that they're sticking with the "hire a HIGH TECH contractor" approach rather than the "hire some coders to work inside the business" approach. Compare SAP/Oracle products to internally developed line-of-business software.
> Who says government projects and agencies have to be poorly run?
I'm not sure how to test this empirically, but it's always seemed to me that these are the inefficiencies that emerge with scale and incredibly broad objectives—and that the federal gov't actually does a decent job considering. Remind which company has 320m consumers, annual revenue of 6.7 trillion, and 2.8m (non-uniformed) employees? The federal government is 10x the size of IBM—it can't be agile. That said, tech is in general not a strong suit for gov't agencies—and it needs to be both 1) for planning, analysis, logistics and 2) for dissemination of information to citizens about what the agency does.
This is an argument for refactoring into smaller modules. Decentralization. Local and state government taking on more responsibilities, with the added benefit of forcing competition and choice among citizens vs. one-size-fits-all policy.
Voting rates for local/state elections are awful, and only rise when they share a ballot with national elections. 'Special districts' are a legal fiction helpful for organizing things like sensible fire department jurisdictions, but they've become a way to ensure that there are few or even no voters for a specific issue, and to minimize state oversight of a given budget. Lobbying groups and special interests may influence national bills, but they write state and local bills, often getting their "sample texts" passed completely unchanged. John Oliver has covered all three of those topics, but I could dig up real sources if you actually doubt them.
I wouldn't care to speak to the fraud claim, but surely it isn't controversial that voter engagement is lower - and N dollars of corporate spending buy more influence - when the stakes are lower?
> Why do government projects and agencies have to be poorly run?
Because lots of people, inside and outside of government, derive significant money, power, or both from government dysfunction. A classic case of goal misalignment.
..two lobbyists who managed to find their way onto the five-person panel testifying. They represented the interests of traditional IT contractors, who seem to believe it is their right to overcharge taxpayers for complex computer systems that don’t work. Even though, as one congressperson at the hearing put it, those special interests are likely to view the USDS and 18F “with a jaundiced eye,” they were given the opportunity to seize on the authority issue as a way to cast doubts on the Obama tech surge.
I think that's part of it, but there's also the facet that a lot of people just...don't care. Families, hobbies, etc are all valid reasons to live for outside of your employment. If the majority do just enough to get by and not get fired (which, as pointed out in other comments, is a pretty low bar in the government, as it is in many large large companies), then the system eventually converges into a sort of homeostasis of mediocrity, which is viciously self-preserving [1].
---
[1] these together are pieces of Yudkowsy's revision of Hanlon's Razor: "Never attribute to malice what you can attribute to an enormous complicated System full of conflicting incentives getting stuck in a weird equilibrium. When that weird equilibrium is crushing people in its gears... We see a broken watch and infer a Watchbreaker."
That's part of it. Back in the mid-nineties the idea that the private sector could do government functions cheaper/better really took hold and the result is that whole core sets of responsibilities are almost completed outsourced to contractors. That in turn has made lobbyists and recently retired senior government and military employees wealthy when they join those same contractors.
However, a problem that has been around a long time is that it is difficult to fire government people for under performance so you have some people in positions of power crippling a whole host of IT efforts through their incompetence. That's not so much about money, it's just that the dead wood is hard to get rid of.
This is the default for large human organizations. The differences are merely shades of the same color.
Also, the purpose of any bureaucracy is self preservation.
The government is a complex adaptive system with almost no external signaling from it's environment. It's an extreme example of how large companies behave when they have no external pressures (competition).
Every system is perfectly designed to produce the results that it produces. I know gov't waste is appalling but you should expect it until selection pressures are applied from the outside. Contrast NASA 1969 vs. 1986 and you'll get what I mean.
This the first and presumably most important requirement in you opinion? Interesting.
May I ask, how did you extrpolate it was a property of the team at all let alone responsible for its success let alone the first component of that success?
Disclosure: I'm an engineer at USDS and these are my own opinions.
We hire for experience, which comes in all sorts of shapes and sizes. Based on my personal observations, the team is actually the most diverse team I've ever worked on by a number of dimensions. We've pulled folks from retirement, others who are chronologically younger but have been contributing to Python and Debian for years, those who have uprooted their family. The job is an incredible mix - it requires being able to debug a slow performing database index one moment, briefing a deputy secretary the next, all while empowering the other federal employees and contractors to do great things.
I would be remiss not to take this opportunity to link to our hiring page. https://www.usds.gov/join
I do recognize that there are lots of talented folks that aren't able to move to DC or can't make it work for a variety of reasons. That's okay. There are many other ways to contribute to our country [0] - working for state/local government, volunteering in your community, being a good parent, inventing the next breakthrough, actually using your turn signals, and many more.
[0] Yes I recognize that not everyone here is US based. Though the point actually is probably true for all countries.
As an old guy, I have other committments besides work. Family, non-profits, etc. It's helpful not to have to worry about those things as well as moving across the country for a lot less pay in a job that might not succeed.
But flexibility, adventurousness, low pay were not listed in the parent post, just age.
Indeed, you can easily insert any idea not actually mentioned post hoc. It's especially easy to insert unsupported but widely believed justifications. For that matter, you could extrapolate that dietary habits or music taste somehow result in less time eating and more team unity if you're absolutely set on justifying a weak assumption or policy.
But particularly one would think the fact they they are not all actually young would be somehow taken into account before justifying they claim that it's the premier attribute to enumerate.
On the flip side, republicans try to sabotage the functioning of many agencies so that they can then complain that the private sector would be a better fit. This has been their MO towards the Post Office for longer than I've been alive, and have been diligently working to undermine social security in the hopes that may be privatized as well. It's the 'starve the beast' strategy on a larger scale.
A note on personal experience as a tech contractor at a federal agency: I worked as a contractor at the US Geological Survey (USGS), in the Department of the Interior each summer between 2006 and 2010.
The most useful thing I worked on was a program for calculating air-sea carbon flux: I did the OS X GUI and helped with the core logic (computing air-sea carbon flux is nontrivial). At the time I was appalled that we spent $10k on what could have just been an R package around a wrapper for the C++ core. But since then the program has become a very common tool for ocean acidification research, with ~175 citations, many of these have since been cited in all sorts of climate change policy documents. It's admittedly hard to assess value, but it's definitely more than 10k. And definitely a public good that the market cannot be counted on to deliver.
I also found out at some point that the contractor I was working for (a Fortune 500 engineering and defense company) billed the USGS more than twice what I received (I got no real benefits). The federal employees I worked with detested the contracting company because they knew how much it was skimming from the contract employees, many of whom were full-time and had been there for years. When I figured this all out, I quit, started my own company, and was hired directly as an independent contractor. Unfortunately, this was not an option for most people, as this wouldn't have worked for a full-time, year-round position.
Disclosure: I'm an engineer at USDS and these are my own opinions.
I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.
I'm curious if anyone here from USDS or 18F has deployed Postgres in a FIPS 140-2 environment, and if so what challenges they had. The VA seems to be saying that Postgres should not be used:
(Don't miss the difference in tone on the "Analysis" tabs.)
I'm sure some of that is due to lobbyists, but nonetheless it seems to me that there are legitimate challenges in making Postgres meet FIPS 140-2 requirements. I've been able to recompile Postgres to use the FIPS OpenSSL wrapper, but storing passwords with MD5 is a harder issue to fix, and of course there is no definitive list. Does anyone have some experience with this?
There's definitely quite a bit of PostgreSQL in use in government, so this does not need to be a blocker. As someone noted, PAM auth is a good solution; and I think if you use CentOS or RHEL (Use the latest release please!) you'll end up with a FIPS OpenSSL which PostgreSQL is linked against.
More generally, the VA TRM is not a permanent thing, it's precisely the type of existing process which can be improved with the feedback of on the ground software engineers. If this is a blocker for you, I'm sure folks at DSVA would be happy to help!
Contractor here, not USDS or 18F. I usually see Postgres configured to use PAM auth, often against LDAP. This can it make it easier to meet password complexity requirements (don't get me started) and lockouts after incorrect attempts (same).
I'm pretty sure the only OS that is approved for electronic voting systems is Windows XP, which has obviously become a problem, but a new OS would likely take two or more years to go through the approval process. Ubuntu changes so much in two years that by the time it is approved, it would have lost 2/3rds of its life even as an LTR.
I would assume that the main reason that Oracle is preferred is due to lobbying, but also the fact that they don't really innovate anymore so it is non-trivial for them to provide support and bug fixes for a decade or two.
In terms of security, that is probably exactly correct (I'm a complete security novice FWIW). Way too high of an attack surface area. That being said, as soon as you get into microcontroller territory, you turn it into a niche thing that makes it inaccessible to most. What are the chances of finding talent that can write user-facing code that runs bug free on microcontrollers while simultaneously writing/proving the crypto? Maybe that talent is more common than I suspect, but I certainly haven't seen it. I would be happy with rust/ocaml on a unikernel or rump kernel though :)
Why does dropping the OS mean you have to run on microcontrollers and write everything from scratch? All it means is your software needs to incorporate the OS functions that it needs for interacting with the hardware. You can still use any hardware platform you want and any third-party libraries available for said platform.
Well, yeah, most microcontrollers require bare-metal programming. That's not my point. My point is any device can be developed for that way, and that going OS-less does not mean having to write everything from scratch. After all, that's what the OS is: a bare-metal application that functions as middleware (amongst other things) for the hardware platform.
Also, I know having an OS makes it easy. If you have full control over the hardware platform and environment, and user interaction with the system is limited to a relatively small set of UI objects, you shouldn't need a full OS. Something bare-bones like DOS or FreeRTOS should be fine. Even on the box I worked on that used RHEL we stripped almost everything out. I'm still not sure why they decided to go with RHEL as the base, considering the amount of work that went into customizing it.
Microcontroller stuff isn't necessarily inaccessible. You can target extremely barebones platforms with Rust, and it goes without saying that C runs almost anywhere.
The point is to have absolute bare-minimum technology. There's no user-facing code to write, it should really be as simple as:
- A function that encrypts an int between 0 and N, perhaps in homomorphic fashion
- N-1 buttons (as in, mechanical contacts) that each trigger a subroutine "Encrypt the number X and broadcast it to whoever is tabulating and/or auditing the vote"
This is my reaction too. There are systems with no security flaws, but they tend to be small-surface devoted hardware. Something the size of Windows will consistently turn up zero-day flaws that could be used against voting, whereas plenty of on-metal microcontroller programs have no holes at all.
Frankly, a really reassuring voting system would be the exact opposite of what we have: it would be open to all security researchers and consist of minimal, verifiably-safe code to count vote totals.
What I would love to see is an 18F-like organization that creates open source software that helps user-facing government organizations at the state and local level. All of these things should be simple but (typically) are not:
* Paying a utility bill online
* Registering to vote
* Finding out how to get from place to place using public transportation
* Paying for public transportation passes
* Signing up for unemployment benefits
* Proving residency
* Getting an ID card
* Registering a vehicle
* Paying taxes
* Entering pleas to local courts for minor offenses and infractions without taking a day off work
* Reporting non-emergency code violations (like construction noise after hours)
* Knowing who your representatives are
* Communicating with your representatives when you do know who they are
We actually use and interact with local and state governments far more than we do with the federal government. And yet when it comes down to it, they are the most difficult to work with because their resources are so constrained. I honestly believe that local governments could benefit from some resource allocation on their behalf at the federal level for the creation of highly scalable technological public goods that will only ever be used at the local level.
I'm an engineer at Code for America - https://www.codeforamerica.org/ - a non-profit that does this kind of work with cities, counties, and states.
Its still pretty early days for civic technology. I think we'll see tech provide all the services you listed, eventually. It'll probably be a mix of outside vendors and local governments running their own engineering teams. There are still some legal barriers in the way, procurement for example.
Its happening, just slowly.
If you are interested in speeding up the progress of your city, I'd recommend checking out if there are any civic tech volunteer groups near you. https://www.codeforamerica.org/brigade/
Disclosure: I'm an engineer at USDS and these are my own opinions.
I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks):
"I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country."
"If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check."
USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript)
A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.
Granted this is one anecdote, but in my experience state and local governments in North Texas are generally much easier to deal with than the federal government. Some are better than others, but I haven't run into any yet that are really bad. Some, like the Dallas Police Department and Dallas Area Rapid Transit, even have mobile apps to facilitate activities that are often done from a phone but awkward to do via the web.
It's heartening to see some new approaches government IT actually getting high level support to make a positive impact. Funnily enough I see someone I used to work with in one of the pictures.
Hopefully this will lead to more substantive change to how government IT is done. It's great to have teams that can drop into problem areas with a clear mandate to get things moving, but the whole system needs some serious reforms so that these IT quagmires don't get created in the first place.
This article heaps a lot of blame on contractors but fact of the matter is, contractors are often hamstrung by what the customer wants them to do and what the customer allows them to do. And some of that comes back to what was specified in the original contract.
Ultimately there are multiple components to the abysmal state of government IT projects:
* The government has moved to a model where very few full time employees are technical people. The idea is that almost all IT will be outsourced to the private sector
* Because fewer and fewer FTE's are technical, procurement and project management decision-making is compromised.
* There is little accountability for government employees, and by extension, contractors. Botched projects where millions of dollars are waste rarely have major career consequences for FTE's, and the bigger contracting companies don't have their ability to get new contracts affected at all.
* Building software for government often involves arduous interpretation of regulations, trying to build to undocumented and byzantine business processes, or waiting an inordinate amount of time for the bureaucracy to make important decisions (then change their mind!).
I feel like empowering the technical people is a very important first step, and getting them executive backing may create the momentum needed to change some of the common pitfalls. But I think there are some big changes that have to happen to the whole ecosystem to really get things to where they need to be.
It's all well and good to give small teams exceptional power to get around the bureaucracy but that approach doesn't fix the underlying problems.
Disclosure: I'm an engineer at USDS and these are my own opinions.
You make a great point - USDS is just one part of the solution. In fact, in almost all of the USDS projects, we work very closely with agency employees and contractors (many of which are just as talented and have chosen to serve their country). Most of the times, I spend very little time hands-on-keyboard and helping empower the existing team.
As for the longer term solution, there is a less publicized version of what folks are doing. USDS has a number of contracting officers who are helping teach others in government how to be savvy customers of technology. 18F and GSA have been doing a lot on this front as well to help bring in really good contractors and writing agile purchasing agreements. The Office of Federal CIO is rewriting and simplifying tech policy and the Office of Federal Procurement Policy has been working on the procurement side. These are the long term changes that I'm personally excited about.
That is great news, I think engaging and training COR's across government could yield some real improvements to how these projects go.
I'm personally seeing some 18F projects come down the pipe that are truly innovative and hopefully will encourage organizations to think differently about IT procurement. Still a lot of silly stuff out there but at least the ship is starting to change heading.
The contractors are hamstrung by their own business model. I'm sure a lot of the people on the ground are trying to do a good job, but when you're got a set-in-stone spec negotiated by people 4 levels of reporting away from the actual work in the trenches.. yeah that's probably not going to be a spec to do the right thing.
Whether they're contracted or government employees, we need to devolve authority from big top-down specs and towards programmers working alongside the actual users.
The problem is that the contractor doesn't write the RFP or the final contract, the customer does. So if you want that money you sign on the dotted line and hope for the best.
Within the contract there is often flexibility as to how the job gets done. But bottom line is that if you have a stubborn customer who insists on doing things the hard/stupid way, there's not a lot you can do. Maybe you go above their head and try to force a change but that is fraught with risk. The prudent thing to do from a financial standpoint is just put your head down and complete the contract, even if the work is compromised because of the customer you're dealing with. And indeed, that's what happens a lot of the time.
Fortunately more and more parts of government are buying into the agile mindset but there's still a lot of outdated thinking out there.
> But bottom line is that if you have a stubborn customer who insists on doing things the hard/stupid way, there's not a lot you can do.
This is something most people who have never dealt with the US government don't understand, thus they put all the blame on the contractors. Many groups within the government are the epitome of the "customer from hell". They hire you to do a job, to build a product, but think that they (and their experts) know better how to do it than you. Essentially, all they want is a body shop to implement their design. This is not what most commercial outfits, particularly SV-type companies, are used to.
In addition to providing technical expertise on the customer side which is hugely helpful, there is also a program from 18F focused on digital acquisition that is helping change the way technology RFPs are written that I think will have a huge downstream impact for any group trying to win the work (internal, or otherwise). The better the customer understands the value and roadmap of the work and can put that into words, the better the outcomes will be: https://18f.gsa.gov/tags/digital-acquisition-accelerator/
The contractors are hamstrung by being contractors. Because the contracting process is adversarial, and geared around a fixed objective against which fixed lowest bids can be made, the kind of iterative development which produces great results is ruled out.
Unfortunately, there's quite a bit of misinformation in this article.
The author makes it sound as simple as hiring fresh hackers from Silicon Valley to take over failed projects from stodgy software developers from the large D.C. contracting firms.
Having worked on government projects like those cited in this article for several years, I can tell you that it is the government agencies themselves, and not so much the developers who are to blame. To build a cool iPhone app for a private service, you can pretty much use whatever tools and timetables you want.
Not so with government tech. With even small projects, you are beholden to policies that were written several levels up. Things like no cloud-based hosting, no sites that don't use government-written style guidelines, and in many cases, no development that occurs outside the government facility. If you hamstrung pretty much any private tech company with the same restrictions government contractors have to deal with, I can guarantee you they would look a lot different.
That's not to say that change can't occur, but to attack the contractors is to misunderstand the root cause of why government software is historically so bad. There are actually many, many very talented developers and software engineers steeped in the latest lean startup theories inside the beltway, but they can seldom use any of that because government policies stand in their way. For most projects, you are dealing with literally thousands of regulations throughout the development process, and most software apps have to account for ever-changing laws and policies that are specific to each application.
However I have already seen a great deal of positive change, with many agencies opening their minds to newer, faster methods of development and allowing better tools to be used.
Not an accident- selectivity bias. Ask those hackers to build an app for the USPS, FDA, the DOE or any other agency that imposes strict regulations and has to implement business logic to account for tens of thousands of rules (not an exaggeration) and see how agile and innovative they are. There's a reason that most of these contracts are worth between 7 and 8 figures. That's the kind of manpower required to engineer them.
The point is that you went from v1 to v2, and the major thing that changed was the age and attitude of the team. If that sort of dramatic improvement was only conditional on other aspects of the situation, fine, but that doesn't invalidate the article.
Great article. For me this is the money quote: “The culture is very persecutory. If something goes wrong it’s usually followed by a witch-hunt, and the result is very chilling. People are afraid to take risks.”
This applies top to bottom (and has nothing to do with the current rage between republicans and democrats). The 1400 pages of travel regulations are a classic symptom.
Disclosure: I'm an engineer at USDS and these are my own opinions.
That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile.
The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering.
This is not a job for everyone. But if you're a certain type, there's nothing quite like it.
The government could do a lot more by adjusting the differentials for technical competence on the GS scale to be somewhere near market (up and down). However, assessing technical competence isn't exactly straightforward.
As I mentioned in another thread, I work for state government and our IT salaries are capped at a max of around $87k/year (most positions are capped lower). As a result, it's extremely difficult to hire someone who isn't a net negative to the team.
Since we have no power to adjust salaries, we're looking at other avenues, such as doing more "direct from community college" in-training hires as well as non-traditional employment paths.
Unlike USDS and 18F, we don't even have exciting projects, a shortcut around bureaucracy, or cutting edge technologies to sell with. The only sales tool we have for candidates is "if you complete your 6 months probationary, you can essentially never be fired or laid off." Sometimes, that attracts the wrong sort of candidate, as you can probably imagine. :)
True, but I wonder what the "normal" rate is. As in, do they really get that much, or by stating that rate on their website are they really just trying to say "if you make a lot more than this and care a lot about money, don't bother".
Even if I made more than that in the private sector, I would be willing to take a minor pay cut for this sort of position. If you think about it, you're helping put taxpayer dollars to more efficient use, which includes your own.
Not sure how many people are going to take a 10% pay cut in hopes that 5 years from now their taxes might be a few bucks cheaper (which of course they won't be because the money would just be spent elsewhere).
Admirable intent, I'm suspicious of its prevalence.
They call it a tour of duty for a reason. $163k would be a massive paycut for me but I'd do it and happily burn savings to make a difference for the lives of my fellow veterans.
Love your second sentence, but really wish they would stop calling it a "tour of duty". It's still a pretty solid amount of money (nevermind Federal benefits) but to really transform we need people willing to work over the long haul not just parachute in for short terms.
Also (slightly biased of course) I bet you get a bit addicted to the mission and that provides a non-monetary incentive to stick around a bit longer :)
It's referred to as a tour of duty because most of us are on 2 year appointments. At the end of that 2 years, there's an optional 2 year extension. These orgs haven't been around long enough for anyone to run out their second two years yet.
Personally, I love this format. It never would have even dawned on me to consider applying for a career position in the federal government, but when the job is time-limited, there's some decision making bug in the brain (even though I could have applied for a normal career position and then just left after 4 years!).
It's a great way to entice people to apply who never would have before.
We also need people in for the long haul. But lets solve one problem at a time!
I love the format as well, and think it's a great idea. I'm currently on a two year term right now at a different org. The two year thing solves a lot (though non-status positions make it hard to hire for the longer term). I'm not objecting to that and agree that we have to start somewhere.
The thing that bugs me is more pedantic. Calling it a "tour of duty" immediately draws a comparison to a military tour of duty undertaken by our military which implies hardship, sacrifice, etc. I get that some people might be taking pay cuts to join or that it might require leaving your comfort zone, but I'd prefer that some sort of sacrifice on the level of heading to Iraq for a year without your family is not implied in the naming.
Honestly, that format is one of the things that attracts me to the USDS (yes, I applied earlier today). I'd like to help some of the obvious inefficiency in government, but I don't want to commit my career to it.
Depends. My wife only has the $9000 in her Thrift Savings Plan after four years at the White House. She was a "political appointee", though, so she didn't get normal GS benefits.
Another project involves working on the Global Positioning System. Lynch still can’t believe that he can work on something that cool while serving his country. “I didn’t even know that we ran GPS, right?” he says. “That’s crazy shit.”
It is funny to explain to people that don't know anything about GPS that the government not only runs it, the Department of Defense is the steward of it. I guess people assume that something as "cool" as GPS system can't be owned and operated by the US government.
I'm excited, skeptical, and interested all at once.
When the hackers leave, who's going to end up doing maintenance on these projects? It's not unusual for a gov't developer to work on a project just long enough to be productive, only to have them leave for greener pastures (often a promotion or relocation). In this case, most will want to stop suffering at lower salary and go back to SV. So it seems like it's going to be USDS holding the bag; unless there's a solid plan to transition the workload back to the VA (either organic or contractor), then USDS will have to maintain every project they start (forever).
On the other hand, I bet there are more than a couple good developers within the VA that would love to take over the projects, if only there were a permanent position they could do it from. Why can't Ash Carter set up more permanent positions?
Worst case scenario, a new President comes in and sunsets USDS, requiring VA to maintain the workload. VA balks, they hand it to Deloitte/BAH/etc, and we're back to square one.
Best case scenario, USDS build quality solutions that last decades and can become the next legacy system that VA builds on. And maybe USDS (if it's still around) can re-revamp the system, etc etc.
Also, I really don't understand the lobbyist's angle on open source: isn't all code written by a federal employee already in the public domain? (with obvious exceptions re: security etc)
18F’s and USDS’s heavy reliance on open source and custom coding raises serious concerns. While open source is an entirely appropriate option for agencies to consider when looking to acquire software solutions for mission needs, we need to ensure that these are not the only options available as different missions have different demands. Making open source software the only preferred solution doesn’t adhere to the tenets of tech neutrality. It also limits options and competition, which means we may not be achieving best value for the taxpayer.
Mmmm, I've been through the process and gotten an offer, and it wasn't that far off what the industry standard is (and honestly beats regular DC rates). They're really solid overall.
Granted everyone's got their own experiences, so YMMV I guess. Just curious where your data comes from?
I read (sorry I don't recall where) a comment on GS levels for USDS. It was like half my current income. Perhaps I should just apply and find out for sure though.
I worked on a government contract years ago. The government employees always told me that their pay was better than it seemed because of the amazing benefits including a great vacation policy. I never made a spreadsheet to test this assertion
how cool would it be if this obama tech team really bolstered recruitment of more software engineers into gov't?
I'm talking federal, state, and local level. Just like each state has senators and reps; 1 engineer for every 1M residents that live in the state.
each state's teams would tackle problems with software, share solutions to widespread, formulaic issues with one another. it'd be a iconic and fresh-breath-of-air kind of way to speed up efficiency and effectiveness (government and big corps are SO SLOW)
This article does a good job of showing it, but everyone talked about is seriously amazing. I made it through all the interview pieces and ran into some other life complications so wound up not getting to work with them, but having been by their space in the Pentagon they're doing some of the coolest stuff imaginable right now.
If you're looking for a new job, I'd highly recommend giving it a shot. Definitely felt like an opportunity unmatched elsewhere right now.
Great to hear Ash Carter is so behind this. That makes me want to sign up for the Defense Digital Service, assuming they have work that isn't frontend-heavy.
> These entities cannot continue to operate as both buyer and seller. There is an inherent conflict of interest when the entity that consults on how to solve a problem then also offers to sell you the solution. The government outlawed such arrangements for contractors, so they should not be permitted to create such a dynamic internally.
That's a very strong accusation from Hodgkins (linked blog post in the article).
There's something about the second photo showing their 'funky' office. That looks like the team member was stuck in an unfinished basement (evoking memories of Office Space).
In hindsight, this shouldn't be too surprising. What might be surprising is that the same logic should apply to ALL government functions, not just software development. Who says government projects and agencies have to be poorly run? Who says we can't devise better systems and/or mechanisms (e.g., market-based mechanisms) for attracting and retaining great people in government?
Throughout history, there are examples of government bureaucracies that have been reasonably well run, in some cases for centuries. The Habsburg empire, in particular, comes to mind: http://www.independent.co.uk/news/world/europe/habsburg-empi...