In my previous two posts I mentioned that I was thinking about the test procedures I'd be using for the rocket motor and, in particular, the "fail safe" aspects of testing. I believe this is an important aspect of testing/launching which, while it is sometimes mentioned in other people's threads, I've never seen it discussed in depth, so here are some of my thoughts--maybe it will be of use to others who stumble on this thread in the future....
Some background:
Every year or two it seems like I read about a "close call" experienced during launch preparation of a rocket--often a large rocket. A year or so ago, for example, someone was filling a hybrid with nitrous and the rocket ignited the moment the nitrous valve was opened. Details haven't been forthcoming, but apparently no one was hurt. Further back in time, someone was arming the ejection system of a large rocket and it went off; again, by luck, no one was injured.
One thing experienced rocket guys do is always scrape the igniter leads together before attaching them to the rocket -- if we see sparks, we know that things are "hot" when they shouldn't be. We do this even though the launch officer has declared the launch system is "off" and even though warning lights are saying the system is "safe". That's because switches can fail, relay contacts can get welded together, and warning lights can burn out. Every now and then I read that someone was "thankful that they did that".
Amateur rocketry is relatively safe and it's because of an abundance of caution that it is. I mention all this because it is the reason for the extreme paranoia running through my thinking about the test procedures I'll be using.
First, some general principles:
One, the test (or any aspect of the test) should not start accidentally. This sounds like an easy thing to arrange, but when complex electronics are involved, it's not as trivial as it seems. It also seems like a really good idea to have more than one manual override during the test startup process. That is, first you "arm" the system, and then you press a "start" button--that way two things have to go wrong for there to be an accident.
Two, in addition to the automated ways of shutting down the test, there needs to be a manual override in case the electronics goes haywire and either can't shut it down or can't recognize that it needs to be shut down.
Three, things should fail "safe". That is, when things go wrong (say, loss of battery power), the dangerous stuff (like the flow of nitrous) should shut itself off. The two most reliable ways of doing this are with springs and with gravity. Both are pretty reliable, but springs are easier to work with.
It took me a while to figure out how to think about all of this, but I finally realized that I needed to have a mental model of the steps in the test process along with a (growing) list of failure modes. Here's an *abbreviated* list of the steps in the test (the full list is over a page long):
0. Assemble the test stand, go through a manual check list of what should be connected and the positions various switches/valves should be in. In particular, make sure certain switches are in a "safe" position so that if the electronics powers up improperly, nothing bad can happen.
1. Power up the electronics; the electronics perform various checks including that various valves/switches are in the "safe" position.
2. Once the electronics is working properly, manually turn switches/valves to the "unsafe" position at the test stand, then walk to the remote location
3. At the remote location, electronically arm the system by turning a switch: the main nitrous valve opens, but the throttle valves stay shut
4. Start the test by pressing a button: under electronic control the igniter starts, throttle valves open
5. If no ignition, stop the test and "safe" the system (meaning make it safe for a human to approach the test stand).
6. If anything else goes wrong, stop the test and safe the system
7. When test is finished, safe the system
Hopefully "stop the test" is done by one or more of the micros, but if necessary the human at the remote location can push an "abort" button that stops the test and safes the system.
For each of these steps, one then looks at all the "stuff" (batteries, valves, switches, electronics)
and asks "OK, what if that piece fails during this part of the procedure?" You need to figure out how you (or the electronics) will identify that it has failed and then figure out what you (or the electronics) will do about it. These *two* steps (identification, response) are both necessary: you (or the electronics) can't respond if the problem isn't first identified. A lot of thinking has therefore gone into answering the question "OK, if X fails, how do I know?".
But here's the "fun" part--lots of electronic components fail "on"--by that I mean that if an electronic switch (like a FET) "burns up" it will often fail shorted so the current keeps flowing. Generally this isn't what you want--you'd like it to blow like a fuse does and open the circuit (thereby shutting off the current). Also, sometimes there are "glitches" during startup so that until a steady-state is reached (perhaps in a few milliseconds) current can be flowing when you don't want it to be. Or, a component can fail silently (say, during test shutdown) and then when you run the test the next time, things don't happen the way you expect. And, of course, things like spark igniters are well known for causing electronics to go haywire (see some of my previous posts about that).
My number one concern throughout this thought process (which has taken place over many months) has been the flow of nitrous: how can I shut it off if the electronics or power fails? Normally (say in car racing) spring-loaded solenoids are used to hold the valve open--if the power to the solenoid fails, a spring returns the valve to the closed position. This is a "fail safe" mechanism which is what you want. But I decided to go with a motorized valve rather than a spring-loaded solenoid, so I don't have that safety mechanism. To shut my valve, the power has to be present (no loose or failed battery connections), the electronics has to work, and the valve motor has to work. Now, maybe I'm being paranoid, but there's a lot of stuff in that chain that could fail and I just don't fancy watching things burn out of control until the nitrous runs out.
For that reason, I cooked up an emergency shutoff valve that will be fitted just downstream of the main nitrous valve. Like a spring-loaded solenoid, it will be spring loaded so that it will shut off without power and it will be held open with an electromagnet. Unlike a solenoid valve it is both cheap and capable of large flows. Here it is in its prototype form, with the valve in the closed position:
And here it is with the valve being held open by the electromagnet:
You can see the electromagnet (circular item, upper right-hand corner), the switch that is used to sense whether the valve is open or closed (small black object with long arm, next to electromagnet), and the valve which is embedded in the blue-green epoxy (since it's roughly cylindrical and a pain to mount otherwise). The small black cube to the left is a relay which I didn't want to include but is necessary because the long run of wire from the remote location to the test stand has too much resistance to run the electromagnet directly. This whole thing is run by the nitrous valve controller which I described in the previous post.
Part of the test procedure will be to manually open the emergency shutoff valve while the main nitrous valve is still closed. Once I'm at a safe distance, I start the test, the main nitrous valve is opened electronically, and the nitrous flows. If anything fails, the current to the electromagnet can be interrupted either by the electronics or manually (by pressing the abort switch on the remote controller) and the spring will quickly pull the valve to the closed position.
(I don't like having to manually open the valve--somehow in this world of automation I've created, it would be far cooler to open it under control of the processor, but it turns out that finding a solenoid capable of overcoming the spring force *and* operating over the 90 degrees a 1/4-turn valve has to operate is really hard (i.e., very expensive). I even briefly looked at making my own solenoid but once I priced the amount of wire it would take, I quickly decided to hand-operate the valve and just hold it open with a small electromagnet. Another option was to use an air-operated valve but that got cumbersome because it required a source of 100 psi compressed air which meant a regulator for my CO2 tank, etc etc. After all was said and done, the electromagnet was far simpler and cheaper. Sometimes practicality trumps coolness.)
To complete the safety aspects of the nitrous supply, I will have a switch and a small indicator lamp near the motorized nitrous valve. As part of the startup procedure, the manual checklist says "nitrous switch is in OFF position". Once the electronics powers up, the manual checklist says "verify nitrous indicator lamp is OFF". This lamp indicates whether there is power going to the valve motor, which at this point there should NOT be. If there *is* power, and I were to move the switch to the ON position, the valve motor would start turning while I'm still at the test stand which would be bad. So, first I check that the switch is off, then I power up the electronics, then I check that the electronics isn't somehow supplying power to the motor when it shouldn't be, THEN I flip the switch to ON so that when the electronics does supply power, the power actually reaches the motor. It's still not absolutely safe, but at least several things have to fail in order for there to be real danger. I'm hoping that's a sufficient amount of paranoia :-)
Now that the fail-safe valve is prototyped and tested with the electronics, it's time to tackle the remote unit--that will just be a little hand-held box that has a small display, an "arm" switch, a "start" button, and the "abort" button. Since its purpose is to keep the human informed about what is going on with the test and to allow the human to start/abort the test (and to do so from a safe distance), its design is heavily influenced by fail-safe considerations. Stay tuned....
--Steve
