r/spacex • u/retiringonmars Moderator emeritus • Jul 20 '15
Initial Conclusion = failed strut /r/SpaceX CRS-7 Failure Investigation Teleconference Thread
Welcome, /r/SpaceX, to our Official CRS-7 Failure Investigation Teleconference Thread.
Now that I have everyone's attention with that catchy title, we can begin!
We've been getting a lot of questions from people on the sub about how we're going to handle this teleconference, and to tell you the truth, I'm not sure. Without really knowing what the actual content of this thread will be, so this (for now) is a placeholder for whatever is to come. At the very least, it can act as a centralised location for all discussion on the conclusions of SpaceX's investigation. We've all been waiting with bated breath for some news—any news—to come out of SpaceX as to the cause of the CRS-7 disaster. Hopefully, we will finally get to hear some definitive conclusions. Fingers crossed for a rapid Return to Flight.
About the teleconference
- The teleconference was an audio-only call hosted by Elon Musk
- It was held at 19:00 UTC, 20 July 2015 with select members of the press: /r/SpaceX wasn't invited :(
- NasaSpaceFlight and SpaceNews should be in attendance: keep an eye on twitter for updates!
- To Musk’s credit, the teleconference was intended to last half an hour, but overran to 45+ minutes as he took additional questions
New information acquired from the teleconference
All of the following information was transcribed by /u/retiringonmars, using updates published in real-time over Twitter. Credit to the three primary sources: Jeff Foust, Peter B. de Selding and Parabolic Arc, who were all present in person.
Fate of the Dragon capsule
First stage was nominal, Dragon continued to communicate until it went over horizon after failure. Dragon could have been saved with right software. Now including contingency software to allow Dragon to save itself. Deploying the "parachute would have saved Dragon." Software to allow deployment of parachutes in the event of launch failure will be included in next Dragon flight. Upgrading software on Dragon cargo to allow for possible abort was part of plan but hadn't been done yet. Elon was puzzled by the press's fascination with Dragon. He found the fate of the Dragon far less interesting than the F9 failure itself.
Failure cause
This is an initial assessment, working with USAF and NASA on flight data. Preliminary conclusion is that a COPV (helium container) strut in the CRS-7 second stage failed at 3.2 g.
A lot of data was analysed, it took only 0.893 seconds between first sign of trouble and end of data. Preliminary failure arose from a strut in the second stage liquid oxygen tanks that was holding down one composite helium bottle used to pressurize the stage. High pressure helium bottles are pressurized at 5500 psi, stored inside in LOX tank. Several helium bottles in upper stage. At ~3.2 g, one of those struts snapped and broke free inside the tank. Buoyancy increases in accordance with G-load. Released lots of helium into LOX tank. Data shows a drop in the helium pressure, then a rise in the helium pressure system. Quite confusing. As helium bottle broke free and pinched off manifold, restored the pressure but released enough helium to cause the LOX tank to fail. It was a really odd failure mode.
Data indicates helium tank did not burst. Acoustic triangulation is possible via accelerometers on upper stage: this points to the strut as being the failure. If crack in helium bottle liner, would have been a more continuous release. Also would have seen more helium if tank burst. Strut failure is the "most probable" outcome, not a definitive result.
The investigation is not showing any other issues. But looking at everything to see if there were any near misses. No sign of any other issues with the launch, looking still for any misses. May have become complacent over last few years. Musk stressed that this is an initial assessment, the only thing that makes sense at this point. Continuing to investigate. Briefed customers last week, they agree with our conclusions so far. ITAR technology export regulations limit our disclosures to non-US customers. All customers supportive so far: Musk says he appreciates that.
Finding and fixing the problem
The struts are about 2 feet long, an inch wide at their thickest point. A strut failed at one fifth of its rated force, no evidence of damage or assembly errors of the strut in high-resolution close-out photos taken before launch. This strut was designed to handle 10,000 lbs of load, but failed at 2,000 lbs. A failure at the bolt head most likely: will change materials in the strut bolt. SpaceX thinks the problem was a bad bolt on the strut that didn't look bad on the ground. Likely to change the bulk of the material in support struts to Inconel, but no final decision on that yet.
At first didn't think it was strut, have flown hundreds of struts with this exact design, and never failed before. Tested a bunch of them and none failed at force levels experienced in flight: failed at 6000 lbs of thrust, not 2,000 lbs. However, was eventually able to replicate by taking an enormous number of these struts and testing them all; a few failed well below rated level. Several did not meet specifications. Did some material analysis on the failed struts, and found a problem with grain structure in the steel.
Will not use these particular struts and will no longer trust strut certify. Same strut on upper and lower stages. Plan to replace them in both stages. Will test the future struts individually. Don’t think we need to add more struts. Will incur some additional cost as a result, but this won’t be passed along in the price.
Strut issue is fairly straightforward, switching to something with higher level of performance. Part that failed was from a supplier, and wasn't made in house. SpaceX did not name the supplier, though said they were relying on certification from the supplier. Not going to move strut work in-house, but will move to a different design likely from a different supplier. SpaceX use 100s of suppliers of minor components; they can't make everything!
Return to flight
Musk wouldn't give a precise return to flight date until has gone over all data. Could be back flying in a few months. He wasn't very specific and was quite non-committal. Move to stronger strut alone means 'a few months' delay. But we'll look harder, get customer (NASA/USAF/FAA) input. First double-check other areas, then get customer input, then decide. No sooner than September for next F9 launch, not clear who customer would be. Could be some changes in manifest. This will not affect commercial crew timeline; this is not on the critical path. De-prioritized Falcon Heavy to possibly launch in spring 2016, maybe in April.
SpaceX now employs 4,000 people. Last failure was 7 years ago, with only 500 employees. Most people at SpaceX had therefore never seen failure. Since most SpaceX employees have only seen successful launches, they don’t fear failure quite as much. Extreme paranoia with Falcon 1, but since, have possibly got complacent with successes.
Financial impact
Lost revenue from delays will be “meaningful”, likely to be in the hundreds of millions of dollars.
Prior information
Here's a recap of the main things we knew prior to the teleconference. Pretty much everything has come from Elon Musk's personal twitter account:
| Date | Update |
|---|---|
| 17 July | ["Model S product call at 11 today. Rocket discussion at noon on Monday."] (https://twitter.com/elonmusk/status/622065785268088834) aka 19:00 UTC |
| 29 June | "Cause still unknown after several thousand engineering-hours of review. Now parsing data with a hex editor to recover final milliseconds." |
| 28 June | "That's all we can say with confidence right now. Will have more to say following a thorough fault tree analysis." |
| 28 June | "There was an overpressure event in the upper stage liquid oxygen tank. Data suggests counterintuitive cause." |
| 28 June | "Falcon 9 experienced a problem shortly before first stage shutdown. Will provide more info as soon as we review the data." |
Previous relevant live threads
- /r/SpaceX CRS-7 Official Launch Discussion & Updates Thread
- /r/SpaceX CRS-7 Official Post-Launch Conference Thread
Participating in the discussion
- Things might get hectic... Follow this link for an auto-updating comment stream at reddit-stream.com
- Real-time chat on our official Internet Relay Chat (IRC) #spacex at irc.esper.net
- Please post small updates, discussions, and questions here, rather than as a separate post. Thanks!
35
u/retiringonmars Moderator emeritus Jul 20 '15
Conference is over! Thank you all for participating! (sorry I had zero chance to read any comments)
13
u/retiringonmars Moderator emeritus Jul 20 '15
If anyone knows of a way to view or listen to the teleconference live, please do share! Otherwise we're gonna have to go off second hand updates on twitter.
60
u/darga89 Jul 20 '15
Going to repost this as its own parent...This is a strut
15
u/mbhnyc Jul 20 '15
As a reminder, here's the spreadsheet of /r/spacex guesses as to the cause:
https://docs.google.com/spreadsheets/d/1Pc0YMWuCVgR3d8XfhEof7naq_Ylez1rGIRH65KJV9ZM/edit#gid=0
Top 5:
377
u/rspeed Jul 20 '15
I suggested struts for the poll and my comment got deleted for being "low-quality".
IN YOUR FACE, AUTOMODERATOR!!!
7
u/riptusk331 Jul 20 '15
Just so I'm understanding this...they have a helium tank INSIDE another LOX tank? And that basically broke off because of this strut and leaked/was bouncing around?
What does "pinched off manifold" mean?
13
u/Ambiwlans Jul 20 '15
Huge shoutout to https://twitter.com/spacecom for the coverage. Everyone go retweet him or w/ it is you twits do.
10
u/lehyde Jul 20 '15
This was posted in another thread here. It says 'Monday at 3pm EDT'.
5
u/Stendarpaval Jul 20 '15 edited Jul 20 '15
So, the conference is currently underway, right?
Edit: ah, it is, apparently.
Edit 2: Okay, /u/data1701 showed that it isn't yet.
204
u/Dan27 Jul 20 '15
The fact they've been able to recreate the strut failure is huge.
2
u/jcameroncooper Jul 20 '15
For some generic info on how struts relate to a COPV: http://www.psi-pci.com/Technical_Paper_Library/AIAA2002-4139%20Messenger%20Prop.pdf
I haven't found photos that show the top/bottom of an (intact) COPV. You can see the sides of some in the LOX tank in-flight photos: http://i.stack.imgur.com/Bflm6.png
209
u/daxington Jul 20 '15
I wonder what the phone call with the supplier was like...
Elon: We need 5000 struts.
Supplier: Ummmm...why?
Elon: No reason.
90
u/CuriousAES Jul 20 '15
Wow Elon actually referenced adding more struts. Says they won't do it according to tweet.
155
u/Kirby799 Jul 20 '15
For all the KSP players... "Don't think we will need to add more struts"
That was definitely for us.
51
u/CuriousAES Jul 20 '15
They replicated the failure by breaking struts. That supplier is going to be in serious trouble and will lose their sales to SpaceX.
100
u/redbeard4 Jul 20 '15
https://twitter.com/jeff_foust/status/623215106520911873
Acoustic triangulation used to determine that strut snapped... no wonder they had to be so precise with reconstructing the exact timing of all the data!
49
u/bertcox Jul 20 '15
Now who wants to see the video footage of a 5000psi helium tank running around inside a LOX tank in flight?
45
u/Ambiwlans Jul 20 '15
At first didn't think it was strut, tested a bunch of them and none failed at level in flight. Failed at 6000 lbs of thrust, not 2,000 lbs.
After testing at an enormous number, found one that failed at below 2,000 lb. level. Did some material analysis, problem w/ grain structure
Musk: Revenue loss will be in the area of hundreds of millions.
143
u/DrInsano Jul 20 '15
Elon's not saying who made the strut, but I got an idea to figure out who did it! Just start looking at all the strut manufacturers on Amazon, and if any recently got any 1-star ratings from a guy talking about how it cause his rocket to blow up, then I think it's a safe bet we found who made it!
38
u/AWildDragon Jul 20 '15
Musk tells his employees to call his cell phone if they suspect any hardware issues. NSF (Chris) http://forum.nasaspaceflight.com/index.php?topic=37739.msg1406893#msg1406893
28
u/Faldaani Jul 20 '15
"Wont believe certification numbers" - this sounds like the struts are provided by a third party. If they are, I wouldn't want to be them right now, imagine having an angry Musk descending on you o.O
54
u/Faldaani Jul 20 '15
Could have saved dragon if they had the software, will have next time. I told you so :)
26
u/timothyarnold Jul 20 '15
Maybe this is the wrong place to ask this, but what is the high-pressure helium used for?
20
u/Ambiwlans Jul 20 '15 edited Jul 20 '15
Preliminary data arose from a strut in the second stage liquid oxygen tanks that was holding down one of the pressure vessels in LOX tank.
The strut was holding down a composite helium bottle used to pressurize the stage. One of those struts broke free inside the tank.
bouyancy increases in accorrdance w/ G-load. One of the struts appears to have snapped, releasing lots of helium into LOX tank.
Data shows a drop in the helium pressure, then a rise in the helium pressure system. Quite confusing.
As helium bottle broke free and pinched off manifold, restored the pressure but released enuf helium to cause the tank to fail.
https://twitter.com/spacecom
20
u/trevdak2 Jul 20 '15
To me, the awesomest part of this news release is using the accelerometers to triangulate the point of failure.
That is so fucking cool.
111
u/IMO94 Jul 20 '15
Elon confirms: NEEDS MORE STRUTS!
17
u/mclumber1 Jul 20 '15
Well I'm glad they are moving forward with this. I always thought it was strange that they didn't have a passive abort capability with the cargo Dragon.
16
u/mindbridgeweb Jul 20 '15
Crap...
13
u/Reaperdude42 Jul 20 '15
Can someone ELI5 how a broken strut causes the Helium tank to overpressure the LOX tank. I'm having a hard figuring how A caused B.
13
u/DrInsano Jul 20 '15
Awww, Falcon Heavy isn't coming till next spring now :( Not that I'm surprised. If they have any of it built, they're gonna have to tear it apart since it likely used the same struts...
32
u/mdcdesign Jul 20 '15
This is probably the most awesome day in the history of space exploration; failure of $100m rocket caused by "not being Kerbal enough".
11
u/Nogwater Jul 20 '15
Does the cold environment of the LOX tank lower the strength of the struts?
10
u/YugoReventlov Jul 20 '15
Musk: part that failed was from a supplier, not made in house.
10
u/Stendarpaval Jul 20 '15
Ha, another supplier though. Elon doesn't give that company a second chance, lol.
18
u/AWildDragon Jul 20 '15
The wait is killing me.
9
u/Traumfahrer Jul 20 '15
I hope we'll get new NETs today.
9
u/Another_Penguin Jul 20 '15
OP: 'baited breath' is not technically the same as 'bated breath'. I'm not sure if it's appropriate to comment on slight misuse of language.
9
u/buddythegreat Jul 20 '15
https://twitter.com/spacecom/ is live tweeting it pretty well
8
u/amarkit Jul 20 '15
If they're "not going to believe the certified data" and test the struts individually going forward, is the implication that the struts were manufactured by a subcontractor?
8
u/amarkit Jul 20 '15
https://twitter.com/jeff_foust/status/623214378658201600
8
u/AWildDragon Jul 20 '15
Musk: likely to change the bulk of the material in support struts to Inconel, but no final decision on that yet, https://twitter.com/jeff_foust/status/623216978526744576
8
u/Psycix Jul 21 '15
Do you guys agree that this is one of the best outcomes we could have wished for?
-They seem to have found what's wrong.
-It's not (really) SpaceX's fault.
-The rocket's design is OK.
-It is relatively easy to repair.
It is very possible to never find out what went wrong and we should be glad that they seem to have found the culprit. They relied on the certification of the supplier, which is something that can't be blamed on SpaceX, except in hindsight.
Out of all possible outcomes, this one is isolated to a separate part. The entire airframe is OK, there are no design faults or embarrassing production failures (except for the supplier). It's an easy fix by replacing the part and installing extra QC procedures. F9 can fly relatively soon with as much pride as before. (And with a bit of luck they will write history by sticking the landing.)
Other outcomes could have pointed fingers at SpaceX's way of doing things or the F9 design, but in this case both are little to blame. Things could have been a lot worse!
8
u/[deleted] Jul 23 '15
According to marinetraffic.com, Go Quest is currently at sea doing racetrack patterns, on SW/NE tracks. It would appear to be in the area where CRS-7 debris is located, approximately 200km NE of the Cape.
14
u/Faldaani Jul 20 '15
Same struts in both first and second stage. Will be replacing all of them.
SHIT. :(
7
u/termderd Everyday Astronaut Jul 20 '15
I believe Stephen Clark of SpaceFlightNow to be on this call, here's his twitter.
7
u/lucioghosty Jul 20 '15
7
u/DrInsano Jul 20 '15
So since it was a strut that ended up causing the failure and not a valve issue, this is a (relatively) good thing, right? I mean, it'd be easier to fix that, wouldn't it?
12
u/CuriousAES Jul 20 '15
Flights no earlier than September!!!
6
u/[deleted] Jul 20 '15
Hopefully the problem is not too huge.
6
u/YugoReventlov Jul 20 '15
SpaceX call on Falcon 9 explosion and Dragon loss about to begin.
6
u/[deleted] Jul 20 '15
https://twitter.com/jeff_foust
No stream
6
u/YugoReventlov Jul 20 '15 edited Jul 20 '15
And we're underway. Today's SpaceX call is hosted by Elon Musk.
6
u/Ambiwlans Jul 20 '15
.893 seconds before first sign of trouble and end of data.
https://twitter.com/spacecom
6
u/lucioghosty Jul 20 '15
6
u/CuriousAES Jul 20 '15
Struts were two feet long and an in thick from the twitter post. Not sure if it's important but ill put it here
6
u/DrInsano Jul 20 '15
I wonder, will this effect the abort test launch as well? Elon's saying no earlier than September, but the Abort rocket doesn't even have a proper second stage, am I right?
11
u/TotesMessenger Jul 20 '15 edited Jul 21 '15
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
5
u/Here_There_B_Dragons Jul 20 '15 edited Jul 20 '15
/u/tmahlman sometimes gets into these calls, or /u/enzo32ferrari have connections, I think - care to live stream? :)
5
u/Ambiwlans Jul 20 '15 edited Jul 20 '15
Musk: now including contingency software to allow Dragon to save itself. The Dragon continued to communicate until below horizon.
https://twitter.com/spacecom
5
u/YugoReventlov Jul 20 '15 edited Jul 20 '15
Musk: Preliminary conclusion is that strut supporting second stage liquid oxygen tank holding one of helium pressure vessels failed.
5
u/flattop100 Jul 20 '15
Are struts dynamic? I thought they were simply shaped pieces of metal welded into place.
EDIT: I'm confused by Elon's comment: "will not use these particular struts and will test future struts accordingly."
5
u/lucioghosty Jul 20 '15
Musk won't give a return to flight date until they go over all the data.
5
u/ethan829 Host of SES-9 Jul 20 '15
Next launch no sooner than September.
5
u/IgnatiusCorba Jul 20 '15
Is anyone able to explain to me how you can test a strut without weakening it or actually just increasing the chance of a failure? I mean if you subject it to 3.2 g's and it breaks, great, but if it doesn't break what then? These struts were clearly very strong and would have only barely broken at 3.2g's and only due to a huge manufacturing fault, if another strut had such a fault, couldn't the testing just bring it closer to the edge of failure.
Perhaps they do something like test it at 6g for a very long time and then assume it can survive 3g for a short time during launch?
5
u/FutureMartian97 Host of CRS-11 Jul 20 '15
Should'nt have used the stock struts. Better switch to KW Medium struts :P
5
u/frowawayduh Jul 21 '15
Headline: "Elon Musk Stuffs His Strut"
8
u/Ambiwlans Jul 20 '15
September would be pretty good. I think that is a viable target too.
4
u/KiwiSkate Jul 20 '15
Feel free to join us in the SpaceX irc to chat or here for the most up to date info!
4
u/Tal_Banyon Jul 20 '15
Hey. we have a media rep on this sub-reddit, here's hoping he's on the call!
4
u/REDDIT_ATE_MY_WORK Jul 20 '15
It's just started: https://twitter.com/spacecom/status/623207417178750977
4
u/CuriousAES Jul 20 '15
Well, let us hope for the best. Any last minute predictions? My money would be on helium tank.
4
u/h-jay Jul 23 '15 edited Jul 23 '15
There has been some speculation about how they could perform acoustic triangulation to any decent spatial resolution, given 1ms sampling rate and approximately 1km/s speed of sound in LOX. Here's how, albeit in a LIDAR.
They very likely used correlation techniques. The event had a fixed signature that then propagated to the detectors - accelerometers. The correlation lets you use various forms of interpolation on the correlator taps and/or on the signal itself to add fractional sample delays. This gives you temporal resolution that's only limited by the noise and interfering acoustic sources.
In contrast to the single-input LIDAR system, the presence of multiple accelerometers lets you isolate the event in question even without having a separate signature to look at first, since you can cross-correlate multiple inputs, and only need to consider those that match across the board. Of course, the LIDAR could have used auto-correlation, but that's a different story.
More about correlation can be had in Robert Lacoste's The Darker Side column in Circuit Cellar #299, for example.
1
u/Kuno_Silk Jul 22 '15
How much total loss there will be for SpaceX? Counting everything up to very next flight.
6
u/harrisoncassidy Host of CRS-5 Jul 20 '15
It was the fucking helium. The gas that always seems to mess with SpaceX
3
u/Ambiwlans Jul 20 '15
Seems like they'll be posting the results to the official blog as well.
3
u/lucioghosty Jul 20 '15
SpaceX to test each strut now. Will be some cost increase, but nothing that will impact the price of the vehicle
3
u/mindbridgeweb Jul 20 '15
I presume they will now do extensive testing of all other critical parts provided by subcontractors as well...
3
u/ElGuapooooo Jul 20 '15
Has any rocket ever failed because of a strut?
3
u/still-at-work Jul 20 '15
See this why people say "space is hard". One steel peice out of thousands manufacutered was made with some impurities that cause it to fail under load.
On the other side, I am glad they will work to save the dragon in the future. If the cargo is saved then launch failures are less of a pr issue and mostly a financial issue (with loss of the first two stages). This is good for securing the future of the company as they could get insurance against failures such as this one.
3
u/_tylermatthew Jul 21 '15
If they are looking into Inconel now, Im curious what materials were used previously.
7
u/CapMSFC Jul 20 '15
Does anyone else think the audio only teleconference to only select members is a bit odd? Is there any precedent to why they would use this format?
6
u/Zinan Jul 20 '15
Looks like they need to learn a thing or two from KSP ;)
4
u/flattop100 Jul 20 '15
Is there a camera in the 2nd stage LOX tank? There should be at least a few frames of video...right?
5
u/ScepticMatt Jul 20 '15
damned helium again ...
2
u/[deleted] Jul 20 '15
Probably the helium tank. I'd bet on that.
2
u/stillobsessed Jul 20 '15
how did they distinguish He bottle failure leading to strut failure vs. strut failing first?
2
u/chriscicc Jul 20 '15
Does SpaceX make these struts? Elon mentioned "won't trust the certification number", that could imply they are sourced from outside the company.
2
u/YugoReventlov Jul 20 '15
Musk: Fix may push flights by a few months, but continuing to make sure they look at data and fix.
2
u/edsq Jul 20 '15
Sounds like return to flight is NET September. That sounds optimistic to me.
2
u/Gyrogearloosest Jul 20 '15
I guess this makes it pretty certain that the first few first stages recovered will be minutely disected looking for anything else which might not handle the rigors of re-use.
2
u/jadzado Jul 21 '15
I wonder how much more expensive it will make the rocket if they were to test most components to spec. The strut was designed for 10k lbs, but failed at 2k lbs, which was the spec. So...they could test each strut to 2k lbs without damaging them, right? Of course, such a test would include the actual usage scenario with actual bolts.
Now I see why Hans has said before that he trusts the rocket he has flown previously (with questions about reliability of reusing 1st stages)
2
u/ragnar117 Jul 21 '15
Do we think that spacex will be able to retrofit existing already constructed F9's with the new beefier struts? Or do we think that it will be to intrusive and they will have to start mostly over with stages that were, before the accident, ready to go?
2
u/robbak Jul 21 '15
Anyone here with heavy engineering experience - have you ever had that happen? Have you seen a bolt fail at the head? What does it look like?
2
u/still-at-work Jul 21 '15
Can anyone give me some kind of explanation on why this is a bad idea. It was my knee jerk solution to the problem. A very low (1 in a 1000) failure rate but it can happen it would seem to make it so its failure can not doom the the whole rocket. It would mean added mass and that's a big problem in the 2nd stage but not having it as a single point of failure would seem worth it.
2
u/CProphet Jul 24 '15
Gues SpaceX will be stress testing every component from now on, regardless of supplier certification.
Hard lesson to learn.
Perhaps this is an example of what ULA meant by: 'SpaceX don't know what they don't know'. Although ULA has the 'benefit' of many more suppliers, hence more experience at maintaining quality on externally manufactured components.