At bases near and far, Air Force coders are revolutionizing the pace of upgrades and development. Illustration: Mike Tsukamoto/staff
It started as a gamble in 2017: Fresh from ditching an over-budget, behind-schedule software contract, USAF set out to learn to code just like the commercial tech industry.
The Air Force called the project “Kessel Run,” a nod to Han Solo’s speedy mission enshrined in “Star Wars” lore. Air Force officials visited Silicon Valley to learn how to apply agile development and operations, or “DevOps,” two years ago, and now the service’s agile coding gospel has spread to the F-35, space systems, mobile apps, maintenance depots, weather forecasting tools, and more.
“Star Wars” and “Star Trek” references abound among the blue-suit coding centers: Kessel Run in Massachusetts, Kobayashi Maru and Section 31 in California, BESPIN in Alabama, Space Camp in Colorado, LevelUP in Texas, and Rogue Blue in Nebraska. In each place, T-shirt-clad coders—some of whom trained at Kessel Run and then brought their new expertise back to their home bases—push out incremental software releases, sometimes multiple times a day. That’s light speed in a world where it used to take years to deliver new features through traditional block procurements.
Now the Air Force’s young software development enterprise faces a critical juncture: USAF’s coding centers need manpower, steady funding, and continued momentum to turn early achievements into ongoing success.
Advocates envision revolutionizing the way systems are developed and upgrades delivered. But others worry that the new mindset may be harder to police and will be stymied by cultural resistance.
Capitalizing on its momentum, Kessel Run’s wins are helping lay the groundwork for others to follow in its footsteps.
“We’re trying to pass this information to anyone else that is trying to stand up an organization so that they’re not starting from scratch,” Kessel Run Director Lt. Col. Enrique Oti said. “They can learn from our mistakes and hopefully be able to launch a little faster.”
Steven Wert (l), digital program executive officer, and Will Roper, assistant secretary for acquisition, technology, and logistics, attend a Kessel Run presentation on custom software applications. Using commercial development practices, USAF is speeding up the pace of upgrades. Photo: Todd Maki/USAF
KOBAYASHI MARU
Like Kessel Run, the Air Force’s first West Coast coding factory was born of a need to turn around a program headed in the wrong direction. Building off the Joint Space Operations Center Mission System (JMS) program, the California-based group is building space command and control and situational awareness tools faster than before, working closely with operators to deliver capabilities based on the most urgent requirements, and adding new requirements as they arise.
JMS began converting to iterative development in August 2018, and the Kobayashi Maru team had completed two increments as of June, according to Col. Jennifer Krolikowski, senior materiel leader for space C2 at the Air Force Space and Missile Systems Center. Each increment, or sprint, includes a specific set of features and lasts about 90 days. Some install technologies created by commercial industry, while other features are coded in-house.
“We’re really trying to capture the commercial companies that maybe typically didn’t work for the DOD but do some really great software work and have some really good brainpower on some of the wicked problems that we have,” Krolikowski said.
These projects can be broken down into four pieces, which might be best understood by thinking about a smartphone. There’s the platform (the phone itself); the application layer (your mapping app, for example); the data repository layer (the information that powers the map); and the infrastructure layer (the phone service provider). Each piece can include multiple products, letting the Air Force work with a variety of partners, instead of choosing one to build everything.
In some cases, software teams are automating and modernizing old code. In others, they are developing new apps from scratch. Projects will expand further into integrated tactical warning and attack assessment, as well as space object tracking.
Among the first products delivered is an app that automated a process that used to take a staffer four hours each day to complete. “By us providing this application,” Krolikowski said, “they were able to bring that down to five minutes.”
“Space Camp” in Colorado Springs, Colo., is an offshoot of Kobayashi Maru. It is building a platform based on open-source code that can host apps, including both unclassified and top-secret, for the SMC. Another offshoot, “Section 31,” brings Air Force and industry personnel together with software companies who can contribute their own widgets.
“Each product line is focused on a specific capability, and then as they’re ready to deploy, it goes,” Krolikowski said.
That gets at the heart of agile development: warfighters and coders decide where the biggest problems lie, then chip away at those issues bit by bit, instead of waiting for the perfect new system to be totally complete. New code is fluidly developed, tested, and then deployed; requirements are set based on immediate and evolving needs as well as whether the group has the resources to address them.
Silicon Valley-based software firm Pivotal provides a development platform for coders and helped prove Kessel Run could work. Now, the company continues to train airmen on rapid software-building techniques they can take to the newer coding factories.
“The great thing about Kessel Run is they helped give a language to a lot of this stuff,” Krolikowski said. “The whole agile concept has been around for a good 10 years in industry. It just took a little while for people to start socializing it in the government side, in the military side.”
Named after an impossible training exercise faced by cadets in “Star Trek,” Kobayashi Maru aims to evolve into the coding factory for all of SMC. Its original C2 products are the first steps toward that goal. Others in SMC are already interested in lessons learned, and Krolikowski envisions working alongside Kessel Run to develop products supporting multi-domain command and control. “That’s something we can offer for the Air Force collectively,” she said.
“We actually integrate and talk with each other pretty frequently,” Krolikowski said of Kessel Run. “I use some of their platform already. We’re both on a path to make sure that we are scalable.”
Startup culture and design thinking contribute to faster software development at BESPIN, a USAF development team at Maxwell AFB, Ala. Here, the mobile-app development team meets with Silicon Valley software developers to learn how they implement technological advances. Photo: MSgt. James Crocker Jr.
BESPIN
Business and Enterprise Systems Product Innovation, or BESPIN, is one of USAF’s newest agile development labs. Named for a fictional planet in “Star Wars” and launched in 2019, the Maxwell AFB, Ala., group wants to create mobile and desktop apps for maintenance crew chiefs, aircrew readiness, and ammunition crews.
BESPIN’s goals are big and its budget small, with just $4 million now and plans to grow to $15 million in the coming years. The Air Force’s logistics, engineering, and force protection branch is providing BESPIN’s first seed money, with an eye toward tools to modernize maintenance and personnel management.
“I don’t have the luxury of a big program with a big budget line to stand up something cool and sexy like Kessel Run,” Business and Enterprise Systems Program Executive Officer Richard T. Aldridge said. “I’ve just got a whole bunch of little programs. So when I pass the hat around, it comes back empty.”
What BESPIN does have is 60 enlisted coders in three product teams who trained with Pivotal in Atlanta before returning to Montgomery, Ala., and taking up residence in space gifted by the local Chamber of Commerce.
To build its crew chief app, BESPIN is scaling up the functions of a tool created at Davis-Monthan AFB, Ariz., for the A-10, with the aim of making it useful to maintainers working on other aircraft. Other team members are working on a desktop tool for aircrew management, hoping to replace 40-year-old software that tracks personnel certifications and more.
Aldridge says BESPIN’s apps will fit into the Air Force’s “flight line of the future,” where maintainers could use Internet-connected tablets to issue work orders, for example, that wirelessly connect to the main maintenance system.
Other apps might include augmented reality: “You just hold the tablet with the camera to the jet and say, ‘I’m going to work on that part,’” Aldridge said.
“The [computer recognizes the] picture, knows that’s the right wing of the plane, and it puts ‘right wing’ in the database.” Three-dimensional renderings could then show maintainers what needs to be fixed and which tools to use, queuing up instructional images and freeing the crew chief from having to be nearby with explanations.
“Once you get … that mobile device onto the flight line, it opens up a whole bunch of other possibilities,” Aldridge said. “We can save airmen’s time.”
He aimed to complete the first crew chief app over the summer, then follow with an ammo troop app in the fall. The aircrew management app is on hold until BESPIN gets money to fund the project. Other projects that may come next: tools to check in airmen as they come off planes, tools to manage trucking overseas, or perhaps new tools to assist refuelers.
“We’re at the ‘think big, start small’ piece,” Aldridge said. “We can kind of figure out what our processes and tools need to be internally to us, and change our culture internally. But I suspect that once we start showing success, we will have a line of customers at the door—hopefully with money in hand.”
A1C Braylen Bartolotti completes inspection forms during an exercise aimed at wielding rapid combat airpower in austere conditions with a minimal footprint. Software maintenance groups could aid mechanics with predictive maintenance algorithms and automated maintenance logs. Photo: MSgt. Meredith Mingledorff
LEVELUP
Another team will develop and test offensive and defensive cyber tools, starting with the Pentagon’s Unified Platform program.
The Air Force said Unified Platform, a joint cyber operations system, debuted in April. That first software release connects the Army, Marine Corps, and US Cyber Command’s cyber systems to pull data from across the department and help spot possible attacks on DOD networks.
“Before this first Unified Platform capability, the services maintained stand-alone applications, resulting in duplication of effort and, more importantly, reduced cyber threat visibility,” the Air Force said. The group aims to update the system with new releases every three months.
LevelUP is also working to connect some Air Force cyber programs with the joint community, said Brig. Gen. Michael J. Schmidt, program executive officer for command, control, communications, intelligence, and networks.
“There’s a council of colonels that [meets] on a regular battle rhythm that decides what are the priorities. … Then the program office executes,” he said. “That execution might be developing a new capability through LevelUP Cyber Works. It might be providing money to an Army or Marine Corps contractor.”
In addition to Unified Platform, the “Cyber Works” is setting up a secure environment for any program that wants to use it to build new apps.
LevelUP also offers a glimpse of how Nicolas M. Chaillan, the Air Force’s new chief software officer, is shaping the coding factories. Schmidt said Chaillan is bringing prototype programs to LevelUP Cyber Works and guiding the direction of Unified Platform.
“On Unified Platform,” Schmidt said, “the minimum viable product was going to be almost unobtainable.” Then Chaillan stepped in and asked, ‘What do you really need in a minimum viable product’ And so he, hands-on, did that with a thousand stickies on the wall.”
Northrop Grumman, systems coordinator for Unified Platform, is helping stand up LevelUP’s continuous development and delivery pipeline, according to a company spokeswoman.
Rise of the Air Force Coding Corps. The Air Force has established several agile coding centers and charged them with rapidly deploying software using the iterative that fueled Silicon Valley’s software revolution over the past decade. Where those centers are and what they do. Illustration: staff
MAD HATTER
The Mad Hatter software team is tackling F-35 maintenance and its troubled Autonomic Logistics Information System (ALIS). Steven D. Wert, the Air Force’s digital program executive officer, said Mad Hatter delivered its initial two apps to Nellis AFB, Nev., in May.
Those apps “help the maintainers do things that they were doing outside of ALIS anyway,” Wert said. “They were using Excel spreadsheets and handwritten notes and then having to re-enter those things.”
Future tools will work directly with ALIS, with the goal of moving all ALIS data into an accessible cloud and pushing out updates faster than Lockheed has been able to do. Air Force Acquisition Executive Will Roper, a top official encouraging speedy coding, has said aspects of the program could apply to F-22s and older platforms as well.
MAINTENANCE DEPOTS
Garnering less attention than the big-name coding factories are changes underway at depot-level software maintenance groups.
Those groups at air logistics complexes across the country are coding centers, but have lacked the agile development model, Wert said. That’s starting to change. At least one, the 309th SMG at Hill AFB, Utah, has already been renamed a “software engineering group” to reflect its evolution.
“What you see now out at Hill, particularly on personnel recovery command and control, is actually the process implementation, the tools, and automation,” Wert said. “You almost could characterize them as a software factory before. I think each of those has over 1,000 software coders.”
The changes could aid mechanics as they experiment with predictive maintenance algorithms and learn to care for increasingly complex, information-heavy assets. A group at Tinker AFB, Okla., is developing mission-planning code for the B-52 and is upgrading software for the B-1, B-2, E-3, and E-8.
“There’s a lot of talent in those software maintenance groups,” Wert said. “As we bring them in earlier on programs, I think there are going to be cases where they’re actually heavily involved in development not after the fact, telling them to sustain software that somebody else has built.”
RAPID DEVELOPMENT, RAPID LEARNING
Kessel Run and its counterparts are still learning. “At its core, we provided that psychological top cover that it’s OK to do this, and it’s OK to challenge tradition,” said Kessel Run’s Oti.
They’ve led the way in experimenting with commercial products and mindsets, shifting how the Air Force tackles cybersecurity and testing, and revamping hiring practices. Kessel Run shares its lessons learned at monthly “enablement days,” and it hosted a “Pitch Day” in July to attract new ideas from the startup community.
The Air Force launched a new “16K” software development officer career field in April, and “8K” for enlisted airmen, signaling that the Air Force recognizes their importance. The new designation’s greatest impact may be identifying and tracking people with the skills and the passion for coding that might not be nurtured in other career fields. Cyber operators were among the airmen traditionally put in some of those roles, but they don’t always possess software skills, Krolikowski said.
In May 2018, the Air Force said it had about 300 airmen who are dedicated software engineers.
The service said in July that it’s difficult to quantify the number of coders, but noted about 3,000 airmen attest to having various levels of coding experience. More than 300 of those airmen have worked in coding groups in the last two years, and about 200 contractors support the three Software Engineering Groups in Oklahoma, Utah, and Georgia.
Will coding centers save money on development? That depends on who you ask. Some say yes, others suggest costs could rise as the Air Force becomes increasingly focused on software, and still others say it doesn’t matter, because doing software like the tech giants is just a smarter way to work.
Krolikowski says she already sees tools becoming cheaper because she isn’t waiting five years before deciding if a product works.
Last year, the Air Force said Kessel Run’s tanker planning tool saved about $214,000 a day in logistics and fuel, and also slashed the time airmen spent in combat airspace. Similarly, targeting tools cut the time it takes to plan out targets by up to 85 percent.
Aldridge said the Pentagon is working with Capitol Hill on a two-year “digital technology management” appropriation for research and development budgets starting in 2021, to try to make funding more predictable. For now, software coding funds are often spread out among multiple categories, making budgeting and planning difficult.
The biggest shift is the idea that capability can always improve, and new features can always be added. “Software never ends,” Krolikowski said. “You never really have a switch over from development to sustainment. There’s not that kind of artificial milestone.”
Aldridge also hopes Congress will allow the Air Force to designate a single enterprise infrastructure on which all other apps can be built, making the overall development environments more efficient. “Those are the conversations we’re all having amongst each other: ‘Does it make sense for each of us to build our own factory, or can I leverage what you’re doing’” he said.
But Oti argues that it would be a mistake to dictate how to create and run coding centers through a central Air Force policy because it could hinder innovation and the natural evolution of these groups.
Wert sees benefits in standardization, however. “I think the nature of the different programs that they’ll focus on will drive some differences,” he said. Additionally, “ I think the standards will be in the most important principles of getting to a release cadence, really leveraging automated tools and automation, and working directly with end users. Those principles have to be there. We will—over time—home in on a set of metrics that make sense, regardless of the differences in practice.”
Over the next few years, the Air Force may settle on a few different models that work well. One could be like Kessel Run, where government leads the work and plays a large role in the product teams, working alongside a talent pool of contractor engineers. Another could look more like a group at Offutt AFB, Neb., where industry comes together to solve problems but government handles the engineering. A third model may take shape in the future.
From Oti’s perspective, that will come as airmen and government civilian staff become more involved in coding themselves. “As more and more government personnel are hands-on keyboarding, building out cloud architectures, building out networks, building out software and writing code, now you have a cadre that’s going to grow up in this next generation of acquisition leadership,” he said. “Which means they can actually make better decisions on technologies.” That, in turn, will lead to smarter contracting and development down the road.
Each software factory is like a startup business. They work through organizational structures, try out different technologies, and experiment with processes until they find what works. For example, at Kessel Run, they tried different-sized teams before deciding that about eight people make up the ideal product team. They also learned not every big idea worked as planned.
But what felt like a gamble two years ago now seems much more promising.
“I know we’ve screwed up in the past. We’ve wasted tons of money on stupid software,” Aldridge said of traditional acquisition programs. But Kessel Run has ushered in a whole new era. “This is brilliant, and it’s the right way to go.”
This story has been updated to reflect that Northrop Grumman is the systems coordinator for Unified Platform.