The most important view of the moon? The one that comes through your dashboard
Firefly’s Blue Ghost director talks telemetry and teamwork
aerospace
We’ve all seen the footage of lunar landings. The Earth’s original satellite looms larger and larger, the craters getting closer, the surface eventually filling the screen, then…touch down.
But that’s not what the mission control teams necessarily sees. And it’s certainly not what they’re focused on. Their eyes will be fixed on the stream of data from every onboard system that tells them what is happening, and whether the mission is on track.
Which is why, for Jesus Charles, director of spacecraft operations for Firefly Aerospace, managing and visualizing that stream of information was critical during the firm’s Blue Ghost Mission 1.
This was the first commercial mission to the lunar surface, and carried a number of scientific payloads, including four from NASA.
Charles’ career to date had included working at the Johnson Space Centre on payloads for the International Space Station, as well as on various cubesat operations.
I saw the crescent, you saw the whole of the screen
Firefly is a “new space” company, meaning it was not necessarily bound by the legacy, largely bespoke technology systems traditional space firms rely on. It could take more open, agile approach overall.
The firm’s production and continuous improvement teams already used Grafana for dashboarding and tracking metrics. When a member of one of those teams moved to the spacecraft team, they suggested it could work for the operations team.
“In my previous roles, when we had to operate space travel, we had to build custom interfaces,” Charles explains to The Stack at GrafanaCon in Seattle.
“It was a lot of upfront work, a lot of development work and validation work that you needed to do just so you could see the data.” Only after that could the team begin to validate the spacecraft. “So I think that's a big difference.”
Firefly's lander is based on “standard” compute platforms he says. “We have a central computer, and then we have distributed subsections, for different capabilities, so we don't have everything riding on just one of the components.”
The lander itself is complex, but the ground infrastructure is also complex. “You can’t have one without the other,” Charles says. “You need both to execute your mission.”
The ground system is mainly based on on-prem ESXi servers in a secure, onsite location at its Austin location in Texas. “They were protected with backup generators and extra measures to be more prepared for any blackouts.”
It also has a secondary system at its launch site in California. “In case of any anomaly, if anything failed, we can jump to the other set of tools and keep the mission going.”
During the mission itself, the lander collected telemetry that it relayed to the ground network, which then went into InfluxDB and then Grafana. This was in the gigabytes range, Charles says. Latency between the moon and the Earth is less than a second, he said. “So as we're seeing the events and the data come in, we know that it's almost real time.”
At the same, another stream of data files related to the ten payloads on board. “So we had a lot of data coming through as telemetry and as files.”
Firefly has a commitment that it will deliver the science data to NASA and other clients within 15 minutes, he said. Over successive lunar missions, “They’re requiring data faster and faster, because we’re all getting smarter on the capabilities that we can execute.”
The front end of all of this is a 15 screen TV wall in the flight operation centre, with Grafana dashboards highlighting telemetry from the onboard systems.
During most of the 45 day mission, an engineer would be assigned to each screen at all times. During the descent, both shifts were on hand, with engineers doubled up on each system – and its corresponding screen.
The descent itself is automated, and while the onboard cameras capture video during the descent, it is not relayed until after the fact.
“So the only way that we can experience this descent is through dashboards and seeing the data on the screen,” says Charles.
And, he says, it was by tracking data that the team could be sure they’d actually successfully landed on the lunar surface via a device which measured acceleration on the vertical axis of the lander.
“So after we made a successful lunar landing, it would need to match the acceleration of the lunar gravity exactly in that axis, showing that we were upright and we were not in a different orientation.” In the event, that’s exactly what it showed.
After March’s mission, Charles and his team are preparing for Blue Ghost Mission 2. This will involve two vehicles – a lander touching down on the dark side of the Moon, with an orbiter to relay communications. In the meantime, Charles and his team will be running through the learnings from the March mission.
“We don't want to reinvent the wheel every time we have a new project, right? The goal is to establish our mission operations capability and just keep that as a standard that we can rely on as we get more projects.”
While Firefly has been able to build systems that are much more open and agile, that doesn’t mean it has rejected the legacy of earlier space explorers. The firefly team was visited by Apollo mission flight director Alumni Jerry Griffin, and also visited other operation centres like the European Space Agency and the ISS.
“We learn from those organizations,” Charles says. Not least from Griffin’s exhortations on the importance of teamwork and the need for resilience.
“Does that mean that we need to do it exactly the same way?” Charles says. “No. But that's the benefit we get, to learn how you approach that, how you have that quality, but you still put it in your new space approach.”