A Typical Morpheus Test Day

by Morpheus Ops Lean, Ian Young (@ICYprop)

Hey Morpheus fans! In this write-up I’ll take you through a test day in a fair amount of detail. Most of you only get insight into the last 20 or so minutes when our Ustream feed goes live. Hopefully you’ll enjoy this behind the scenes look into all the hard work the team puts in to make each flight happen.

Overview

As you probably expect test day is a busy day with many tasks to accomplish to get to ignition. The overall day can be broken up into several different sections, each of which I’ll go over in more detail below. The different portions of the day are Safety Brief & Vehicle Rollout, Pre-Fill Checkout, Propellant Load (Liquid Oxygen and Liquid Methane), Leak Check, Final Preparation, Flight, and Post Test. A typical test day is about 10 hours from roll-out until Morpheus is back in the hangar. There are two teams working in tandem to get Morpheus ready for flight, the Pad Crew and Control Center. The Pad Crew is out at the pad and performs all the physical tasks needed to get ready for the test; this includes flipping switches and hooking up hoses. The Control Center operators monitor telemetry and send commands to Morpheus. (For more details on the various positions see the previous blog post)

Safety Brief & Rollout

The test day begins with the entire team meeting at Morpheus’ hangar, where our PM goes over the plan for the day. The briefing starts by covering hazard and safety topics applicable for that day’s test, such as cryogenics, pressure vessels, and laser firings. It’s vitally important that in a contingency the entire team knows what actions need to be taken. After the safety brief is covered each discipline reviews changes or items of note for their subsystem, including giving their concurrence they are ready for flight. At this point the team is ready to get the test day underway.

The pad crew finalizes Morpheus for transport from the hangar to the pad area. Morpheus must then be transported from the south end of the Shuttle Landing Facility (SLF) to the north end. The 3 mile trek takes about 20 minutes to ensure Morpheus isn’t jostled too much on the ride.

While Morpheus is on the way to the pad, the control team heads to the SLF air traffic tower where the control center is located. Here the operators begin configuring their consoles for the test day.

Once Morpheus arrives at the pad it is lifted off the cart by a crane and placed on launch stands. The three launch stands align with three load cells on Morpheus that allows the team to monitor the weight and center of gravity (c.g.) up until the moment of ignition. The launch stands fall away as Morpheus lifts off. Once on the launch stands, the pad crew then begins positioning all the ground equipment required to get Morpheus ready for flight, items such as grounding straps, propellant tankers, and ground power. When both the pad crew and control center teams are ready, the team powers on Morpheus’ avionics and moves on to system checkouts.

Pre-Fill Checkout

Prior to loading propellants all of Morpheus’ systems are checked out. This allows the team to work any issue that may come up before the cryogenic propellants are onboard. Most of the day Morpheus is powered by a ground power cart to preserve the onboard flight batteries. Once powered up the vehicle is precisely leveled on the launch stands using the onboard Inertial Measurement Unit (IMU). Being level helps ensure that when propellant is loaded it will settle out evenly between the two tanks.

Once level a process known as ‘Gyrocomping’ is kicked off. The gyrocomp initializes the accelerometers inside the IMU and must be performed any time the IMU is powered on. The process is about 3 minutes and internal to the IMU. During this time the vehicle must be as still as possible as the gyrocomp uses the sensed rotation of the Earth as part of its process.

After gyrocomp we checkout all the mechanical valves and actuators to ensure proper function. First, we test out the helium disconnect mechanism. Helium is used to pressurize the tanks to their flight pressure. A remote system is needed because the high tank pressures means the pad crew must be at least 1250 feet away. After a successful disconnect checkout we check all the other valves (which include vent valves, cooling valves, and engine valves) on Morpheus. This is a total of 16 valves.  We then move on to the actuator check. For this portion we put Morpheus on to flight batteries to ensure they can handle the power draw required to drive the actuators simultaneously. There are three elctro-magnetic actuators (EMA’s) on Morpheus; one drives the throttle and the other two gimbal the engine. With those checks complete we command each of the four RCS jets valves open and to spark. Finally, we conclude this portion of checkouts with checking the main engine spark.

In parallel with the actuator checks we checkout two of the three ALHAT sensors, the Doppler Lidar and Laser Altimeter. These two sensors don’t give very useful data until we’re moving, so at this point we are mainly looking to ensure they booted up properly and that the temperatures are all nominal. When the actuator and ALHAT checkouts are complete we go back on to the ground battery cart.

To complete the functional checkouts we test the remaining ALHAT sensor, the Hazard Detection System (HDS), and the Thrust Termination System (TTS). The TTS’s main function is to independently shutdown the main engine should the need arise. The TTS accomplishes this by shutting a valve on each of the propellant feed systems to the main engine. The TTS also has a function to stop the HDS laser from firing, thus providing an independent method to safe that part of the vehicle as well. The HDS system uses a laser that is not eye safe so extra precautions are taken by the team to ensure safety. With the HDS system powered, the full functionality of the TTS system is checked by verifying that it will also stop the HDS laser from firing.  The HDS is then pointed at two different targets to verify pointing accuracy of the system. Precisely pointing the HDS system is a key requirement for the ALHAT flights.

At the successful conclusion of the functional checkouts the team prepares to begin loading propellants on to Morpheus.

Propellant Load

At this point the team is ready to load the Liquid Oxygen (LOX) and Liquid Methane (LCH4) on to Morpheus. Generally, we load LOX first but can, and have, loaded Methane first. During loading we fill to fairly precise quantities. Of course we don’t want to run out of propellant during a flight, but we also don’t want to overfill too much. Any propellant that we aren’t planning to burn is simply dead weight that we have to carry with us. So you can see it’s a fine balancing act. This is where it’s very handy to have Morpheus resting on the three load cells; their weight readings are our best gas gauge.

Because the propellants are cryogenic, they will continue to boil off throughout the day and this must be accounted for by the Prop officer when setting the loading targets. When the propellants are first put into the tanks they boil off at a very high rate because the tanks are at ambient air temperature. Once the tanks chill down the boil off rate is very predictable. To help avoid some uncertainty, after the rapid boil off is finished we top the tanks back off to the target load. Once both propellants are loaded the team is really on the clock. If too much propellant is allowed to boiloff there won’t be enough left to perform the day’s test.

Leak Check

The final checkout that can only be performed once propellant is on board is a leak check. With cryogenic propellants on board, seals and fittings can shrink allowing propellant vapors to escape, so a leak check is performed. Valves are closed to stop the propellants from escaping as they boil off, and so that pressure can be built up in the tanks. With the pad crew safely back from the vehicle the tanks are pressurized up to 40 psi with helium. After a wait period, required personnel are allowed back to Morpheus to check for leaks and make final torques on the propellant systems. The leak check generally takes about 35 minutes to complete. For most on the team this is the final calm before the storm.

When leak check is complete the TTS is checked out one more time at cryogenic temperatures and then the pressure is released from the tanks to allow the propellants to cool back down. (The explanation for that phenomenon is for future blog post.) Morpheus and the team are now ready to make the final push to flight.

Final Preparation

While the leak check was under way the pad crew was busy removing any unnecessary equipment and staging it for when the team retreats. The ground cooling of the avionics is disconnected, Morpheus is switched over to flight batteries, and the various on-board cameras are turned on. Finally, the valves are once again closed to allow for pressurization and the crew is ready to retreat from the pad area. Once the pad crew has safely retreated pressurization of the tanks begins. Pressurization takes about 20 minutes, and the flight pressures range from 300-355 psi, depending on the objectives of the day. As pressures come up we use some of the methane to cool Morpheus’ electronics, and ALHAT is also configured for flight. Once at flight pressure the helium line is disconnected and retracted to a safe distance.

The team is now ready for the final Go/No-Go poll conducted by the Test Conductor (TC). With all systems “Go” and final words from the Flight Manager (FM), the final engine conditioning is performed. The command to start the onboard ignition sequence is sent. 5…4…3…2…1…

Flight

If you’ve seen the video then I think this section speaks for itself. (A future blog will detail what Morpheus and ALHAT are doing during the flight)

Post Test

Once Morpheus is safely on the ground at the landing pad the control team executes the necessary commands to safe the vehicle systems prior to allowing the pad crew to head down range. This means venting leftover pressure, making sure there’s no fires on the vehicle, and that the laser has stopped firing.

Once down range, the pad crew works to put Morpheus on stands, so the control team can get weight readings as propellant offload occurs. If need be, the pad crew will also hook up ground cooling and power.  Propellant off-load happens one commodity at a time to get an accurate reading of how much remained on board. This information helps to refine the engine model used to predict performance. Once offload is complete, Morpheus is rolled back to the hangar where the vehicle methane tanks are inerted for safety, post-flight inspections are completed, and high-rate data is offloaded in preparation for the next flight.

Test Day: Roles and Responsibilities

By Morpheus Ops Lead, Ian Young

Test days are obviously a busy time for the Morpheus team. This blog is to give all of you some insight into who the players are. In a future blog we’ll bring you a narrative of a test day.  

Who makes up the team and what their roles & responsibility are on the day of a test.

Flight Manager (FM) – The Flight Manager has overall responsibility of the test. This is the position that represents project management, usually either the Project or Deputy Project manager. The FM coordinates any external interfaces needed on test day. They monitor the day’s activities and give the final Go/No-Go for the test.  

Test Conductor (TC) – This position is in charge of all the test day operations. The TC focuses on overall vehicle and crew safety, while integrating operations between the various control disciplines and the pad crew. TC also performs all the voice communications between the control center and pad crew. The position can be likened to the Flight Director in the Shuttle or Station mission control room.

Morpheus Control Center at Kennedy Space Center during a Free Flight Test Day

Operator – Assists TC in the day’s operations, and helps ensure all steps are completed as required in the test procedures. The Operator sends all commands to the vehicle leading up to and including the start of the ignition sequence. The Operator also develops the day of flight procedures used by the team. 

Range Safety Officer (RSO) – Staffed by a member of the test safety group from JSC, the RSO has ultimate responsibility that Morpheus stays inside the range boundaries defined at KSC. Should the RSO feel the vehicle poses a threat to any personnel they have the ability to terminate the flight via the independent Thrust Termination System (TTS). The RSO has the authority to terminate the test independent of the operations team at any point they deem necessary.

Test Area Manager (TAM) – The TAM is Morpheus’ liaison to KSC. On test days the TAM clears the SLF air space, and coordinates with the NASA Air Traffic Control tower. The TAM also coordinates other KSC operations, such as crane riggers and truck operators for the cryogenic tankers.

Guidance, Navigation, & Control (GNC) – The GNC console is in charge of designing the day’s trajectory as well as the sensors such as GPS and Inertial Measurement Units (IMU) that keep the vehicle headed in the right direction. The trajectory is developed several days ahead of time so that many simulations can verify the correct parameters are set on the day of flight. On the day of flight the GNC makes final tweaks to the plan to account for test day variability like the actual prop load and the winds of the day. The GNC console must also monitor the GNC sensors for anomalous behavior and to ensure they are properly initialized prior to flight.

Propulsion (Prop) – The Prop console is responsible for all components related to the propulsion systems of Morpheus. This includes propellant tanks, main engine, Reaction Control System (RCS), and feed systems. Based on the trajectory design of the day the Prop console will calculate the propellant load of Liquid Oxygen (LOX) and Liquid Methane. Their calculations have to include main engine run-time, boil-off, and reserve. During the flight Prop monitors the main engine and RCS thrusters for correct performance.

Avionics and Power Systems (APS) – APS monitors the flight computer and power systems. Morpheus has flight battery system that must be verified prior to flight. There are a number of electrical systems that must be monitored throughout the day to ensure a successful flight.

ALHAT – When the Autonomous Laser Hazardous Avoidance Technology (ALHAT) package is installed on the vehicle, this position monitors the health of the three ALHAT sensors (Doppler Lidar, Laser Altimeter, and Hazard Detection System)

Ground Data Systems (GDS) – The control center has multiple workstations the operators use for command and telemetry and these are maintained by the GDS. GDS is staffed by the Flight Software (FSW) group and are in charge of both the ground systems software and the on-board flight software.

Morpheus Comm – These folks are responsible for the command and telemetry radios that relay Morpheus data back to the control center, along with the infrastructure that goes along with it. They also set up many of the cameras that are used for situational awareness inside the control center, as well as streamed out to our live audience.

Pad Crew: There are a number of Pad Crew that prepare the vehicle for test flight.

• PAD 1 – PAD 1 coordinates all pad activity. Throughout the day the vehicle is put through functional checks, filled with propellants, leak checked, and put into final preparations. There are a number of personnel that help accomplish these tasks, including crane riggers, cryo operators, and other Morpheus personnel.
• PAD 2 – Morpheus personnel who assist with propellant loading and leak checks.  
• PAD 3 – This position operates all the cryogenic tankers for propellant loading.
• Structures – The structures representative assists the pad crew with items that fall under their responsibility, such as launch stands and tether configurations.

Greatness on the Horizon

A note from Jenny Devolites, SE&I Lead for Morpheus, after a successful Free Flight Campaign in December...

As everyone who follows Morpheus knows, our first Morpheus “Campaign Zero” free flight test campaign at Kennedy Space Center in 2012 ended with a spectacular vehicle crash, after 26 successful static hot fire and tethered tests. Shortly after the crash, our Project Manager, Jon Olansen, gathered the sad but resolute team at the KSC hazard field landing pad and reiterated, “this is why we test.” We keep the dream, we learn from our failures, and we try again. NASA management, all the way to the top, understood the risks, and immediately turned us around to rebuild and fly again, because they understand the value of what we are trying to achieve. They can do this, because we are a low cost and lean project with an incredibly dedicated and competent team – a team who wants to build rockets and spacecraft and do whatever they can to help further human spaceflight and exploration.

Last week we had our first successful free flight of the “Bravo” vehicle. The flight was impressively on target with regard to the planned trajectory. All of the things we worried so much about – because of the Alpha vehicle crash and all of the testing since then – turned out to not be problems on that beautiful flight test day. There was definitely something redeeming for the team and the Agency -- to show that we could do this.

This past weekend, a bunch of the Morpheus team members went to KSC’s new “Atlantis” space shuttle orbiter visitor’s center. The exhibit is phenomenal, and found many of us with tears in our eyes as we revisited that amazing engineering endeavor coming to a close. One part of the historical film struck me – that the engineers who conceived of the space shuttle set out to make it a reusable spacecraft. Many rocket engines and spacecraft are single-use. Our little vertical test bed – our lander – is also designed to be reusable. But we don’t have thousands upon thousands of team members or a budget like that – we are a small project with less than 50 people (and that’s padding the number).

So just doing one successful free flight would be a career achievement – but two in one week is a tribute to the hard work and efforts by a team that has become like family. Today’s free flight went higher, and further, and faster – and it was 82 seconds of picture-perfect flying.

Last year after the crash, I remember vividly walking to the building next to our KSC vehicle hangar and seeing the Alpha vehicle wreckage for the first time up close. The crash itself had happened so fast that there was barely time to comprehend what happened. But there was the wreck, our wonderful creation, smashed and burned.

Today I waited outside in the sunshine (we’ve been lucky on weather this week here in Florida compared to much of the country) for the Bravo vehicle to be driven back to the hangar. I watched it roll in and marveled at the gift I have been given to be an aerospace engineer on a project such as this one. After today’s flight test we will spend a couple of days inspecting the systems and putting away all of the ground support equipment to get it ready for Campaign One in January, where we will fly increasingly challenging flight trajectories with our prototype lander vehicle.

And So We Begin Again

The roar of a 5,000lb rocket engine has returned to the Johnson Space Center. The Morpheus team has completed the build-up of our “Bravo” vehicle, conducted numerous integrated tests, and has now stepped into our flight test program. We are picking up where we left off – in fact we never stopped working. We have completed our first major milestone in conducting a 50-second static hot fire of the main engine in the vehicle, including simultaneous demonstration of thrust vector control (TVC) and integrated methane reaction control system (RCS) jet firings. Thrust vector control is used to balance and fly the vehicle, while the RCS jets are used to keep the vehicle pointed in the correct direction. We will step into dynamic tethered flights soon, in preparation for our return to KSC this summer.

The knowledge and insight we gained over the 27 test firings of the previous vehicle are fully incorporated into the testing we’re beginning now. Although a hardware failure led to the loss of the original vehicle last August, the failure and our internal investigation gave us valuable insight into areas that needed improvement. The vehicle may look largely the same as the previous version, but there are numerous changes that have been incorporated.  We have now implemented 70 different upgrades to the vehicle and ground systems to both address potential contributors to the test failure, and also to improve operability and maintainability.

Hard at Work

The Morpheus team has been hard at work preparing for this year’s series of tests and building the new Morpheus 1.5B and 1.5C vehicles.  We have been busy assembling the vehicle structures, wiring in all of our sensors, running integrated tests, continuing engine firings at Stennis Space Center, and more.

Hard at Work

Langley Helicopter with ALHAT sensors attached
running simulated Morpheus flights at KSC.
Image Credit: NASA

The Morpheus and ALHAT teams are now a combined team, which enables a more integrated series of tests as we prepare for future flight tests.  One of these integrated tests took place at Kennedy Space Center in December.  We used a Langley Research Center Huey helicopter as a stand-in for Morpheus.  We mounted the ALHAT sensors under the belly of the helicopter pointed in the direction of the helicopter motion.  Other components such as sensor electronics, Morpheus flight computer, real-time communications equipment and support hardware were placed in the passenger/cargo area.  This allowed both onboard and ground support teams to monitor progress in real-time.  The helicopter was  flown repeatedly on Morpheus-type trajectories towards the hazard field. 

Moving Forward, Not Starting Over

"A ship in harbor is safe, but that is not what ships are built for."

-John Augustus Shedd

On Thursday we made our second free flight attempt with the Morpheus prototype vehicle.  As you can see in the video below, shortly after liftoff we experienced a hardware failure and lost the vehicle.  The root cause is still under investigation,  but what we do know is that at the start of  ascent we lost data from the Inertial Measurement Unit (IMU) that supplies navigation updates to the flight computer.  Without this measurement the vehicle is blind and does not know which way it is pointing or accelerating.  Since this data is needed to maintain stable flight, the vehicle could not determine which way was up and began to tumble and  impacted the ground about 50 feet from the launch site.  No one was injured, no property was damaged besides the vehicle and we have been able to recover significant data, which will give us greater insight into the source of the problem.


We have said it before and will continue to say, this is why we test.  We have already learned a lot from this test and will continue to learn as we recover data and evaluate the hardware.   No test article should be too precious to lose.  A spare vehicle was planned from the start and is just a few months away from completion.  The basic development approach is to quickly build, test and redesign the hardware to achieve many design cycles and maturity before building flight articles.

Tether Test 20: First KSC Flight

The Morpheus team successfully flew our first tether test at Kennedy Space Center on Friday August 3rd.  The objectives for this flight, along with the dry and wet runs earlier in the week, were to verify all systems were in good working order after shipping from Johnson Space Center in July and to allow the new KSC support team an opportunity to move through flight procedures.



After looking over the data over the weekend and coming together for a Test Readiness Review, our Project Manager, Jon Olansen, approved our first ever free flight for August 7th.  This will be the first time we will fly the vehicle without a crane attached.  The crane was used in previous tests as a safety mechanism to allow each subsystem to safely tune their individual systems for a smooth stable flight.