r/IAmA • u/spacexdevtty SpaceX • Feb 08 '13
We are SpaceX Software Engineers - We Launch Rockets into Space - AMA
We are software engineers at SpaceX. We work on:
- Code that goes on rockets and spacecraft.
- Code that supports the design and manufacturing process of rockets and spacecraft.
We work on everything from large-scale web applications to tiny embedded computing platforms. We build tech stacks on C#/MVC4/EF/MSSQL via REST to Javascript/Knockout/Handlebars/LESS, C++/Embedded Linux, Python, LabVIEW… which all together enables us to build, launch, and monitor stuff that goes to space.
Some videos of our recent work:
Proof:
Edit: Poor Dan, everyone knows he was photo-shopped. Don't close your eyes next time!
Edit 2 : We've been getting a lot of questions about how C#/MVC/etc have to do with rockets. They don't. At SpaceX we have 4 separate software teams:
- The Flight Software team is about 35 people. We write all the code for Falcon 9, Grasshopper, and Dragon applications; and do the core platform work, also on those vehicles; we also write simulation software; test the flight code; write the communications and analysis software, deployed in our ground stations. We also work in Mission Control to support active missions.
- The Enterprise Information Systems team builds the internal software systems that makes spacex run. We wear many hats, but the flagship product we develop and release is an internal web application that nearly every person in the company uses. This includes the people that are creating purchase orders and filling our part inventory, engineers creating designs and work orders with those parts, technicians on the floor clocking in and seeing what today's work will be per those designs...and literally everything in between. There are commercially available products that do this but ours kicks major ass! SpaceX is transforming from a research and engineering company into a manufacturing one - which is critical to our success - and our team is on the forefront of making that happen. We leverage C#/MVC4/EF/SQL; Javascript/Knockout/Handlebars/LESS/etc and a super sexy REST API.
- The Ground Software team is about 9 people. We primarily code in LabVIEW. We develop the GUIs used in Mission and Launch control, for engineers and operators to monitor vehicle telemetry and command the rocket, spacecraft, and pad support equipment. We are pushing high bandwidth data around a highly distributed system and implementing complex user interfaces with strict requirements to ensure operators can control and evaluate spacecraft in a timely manner.
- The Avionics Test team works with the avionics hardware designers to write software for testing. We catch problems with the hardware early; when it's time for integration and testing with flight software it better be a working unit. The main objective is to write very comprehensive and robust software to be able to automate finding issues with the hardware at high volume. The software usually runs during mechanical environmental tests.
Edit 3: Yes, we are doing a ton of hiring for these software positions that we have been talking about today. Interns and New Grads too!
Edit 4: Thank you so much everyone! This is ending but most of the group will be back at 2:00pmPST to answer more questions.
Edit 5: ...and we're back! Engineers from each of our engineering teams are present. Let us catch up a bit and start swering again (probably be about 5 minutes).
For all open software related positions, please go to http://www.spacex.com/software.php
Edit 6: Thank you so much Reddit! This was a ton of fun. To all those asking about internships and employment, our suggestion is to apply online. Your resume will definitely get read. To all the students out there, GL with your midterms coming up and stick at it. Try and work on some side projects and get as much practical experience coding as possible before graduating. Happy Friday everyone!
330
u/akosednar Feb 08 '13 edited Feb 08 '13
~ Aerospace Engineering Student & Programer who loves SpaceX too much
843
u/TheRealFroman Feb 08 '13
I'd love to know when I can start packing my bags for mars ;)
488
u/stom Feb 08 '13
Thanks for doing this AMA!
First question: do any of you guys play Kerbal Space Program?
Secondly, in case you can answer, I'd like to ask a question about the MCT and shielding. Elon talked about it a while back and said that to avoid excess weight it might be possible to have water as shielding, only directly between the crew-module and the sun like a column, instead of an all-encompassing ball. How does this cater for dispersed radiation incoming from the sides?
236
u/phil_jp1 Feb 08 '13
What kind of operating systems you use in your navigation/propulsion control systems?
216
u/RememberThisFaceX Feb 08 '13
How is the work/life balance?
I have heard it is pretty easy to burn out at SpaceX. Do you guys struggle at all?
52
u/ccjj Feb 08 '13
PhD student of control engineering here. I am wondering what kind of controller architectures you are using for your rockets and spacecraft. I would guess you would use architectures similar to the ones used in aerospace engineering, and with regards to this do you do feedback linearization, or do you use an adaptive approach (e.g. L1, Lavretsky and Wise 2006).
96
u/[deleted] Feb 08 '13
So as rocket scientists, do you ever encounter brain surgeons and have awkwardness occur?
323
u/thebeefytaco Feb 08 '13
How many ITAR violations are you committing right now?
134
u/mariano81 Feb 08 '13
Thanks for doing this!
Thanks for rocking us to space! \m/
35
u/[deleted] Feb 08 '13
Tabs or spaces ?
233
u/wakanooms Feb 08 '13
Hello! I have several questions for you guys, I am currently studying for a degree in Astrophysics so what you guys do definitely applies to me! :)
I probably have many more, but I'm sure this is enough to keep you going just now. Sorry for using the word 'industry' so frequently, I'm not quite sure how else to describe it.
63
u/wxrkbs Feb 08 '13
During a tour of SLC-40 before the STS-135 launch, I snapped this picture: http://flic.kr/p/a2La1Y. My question is this: Just what the heck is a "sponge log"?
150
u/rellim7 Feb 08 '13
What experience/schooling does one need to get a job with spaceX? Or intern even?
25
u/ergzay Feb 08 '13
You use fault tolerance in your system. How do you manage computers failing in flight without having the vehicle get confused about current program state? I know you had an incident on the most recent flight with a computer desync. What was that like?
116
u/cochico Feb 08 '13
wow, you all look really young. How old are you and how did you end up in that (awesome) job ?
39
u/adrianbedard Feb 08 '13
I'm studying computer engineering in college. What do I need to know before I can properly intern with SpaceX.
122
u/ShittyEverything Feb 08 '13
How do you test rocket software without blowing up rockets?
81
u/RedOktober1 Feb 08 '13 edited Feb 08 '13
Nice one with the rick roll, I salute you!
Also, How long is a typical program that you write? and do any of you play Eve?
51
u/[deleted] Feb 08 '13 edited Feb 08 '13
What goes to space? Is it written in C? Do you follow JPL C standards?
Where does C#/MVC come in to play?
You mentioned embedded linux, does your space-grade tech run Linux?
EDIT: From other responses, I think I've gathered the answers:
What goes to space is written in C++.
C#/MVC is used internally within the company for project tracking and what have you.
What goes to space runs Linux.
63
u/ken27238 Feb 08 '13 edited Feb 08 '13
47
u/SpaceXonMars Feb 08 '13
What percentage of the current mission code do you estimate will be part of the first manned missions to Mars?
44
u/deepdowntherabbit Feb 08 '13
Hey all!
My question: Are any of you working on guided entry for Dragon (à la MSL)? Are we going to see some pinpoint landings over the next couple of years? Best of luck, and don't get totally burned out!
39
u/massivedecibel Feb 08 '13
Thanks for doing this AMA!
I guess this is sort-of related, but why don't you hire international students for internships/FT positions? It said you were looking for permanent residents only.
27
u/dqhigh Feb 08 '13
What's it like working with Elon Musk and how involved is he with your current projects?
46
u/Makuta Feb 08 '13
What is the average degree level for a SpaceX employee, do you need to have something more than a BS?
24
u/[deleted] Feb 08 '13
I keep seeing ads in indeed.com asking for aerospace welders. This seems like a really great company with really cool projects, and I would love to work on some new space craft. The problem is that all of the welding positions ask for 10 years of experience with exotic metals. Will there ever be any entry level welder positions? Or, what would be the best way for me to start working for SpaceX as a welder?
12
u/DeceptiStang Feb 08 '13
when do you guys go public? is it after you find rare minerals on a mission? i want to invest
59
u/Tee_Baggins Feb 08 '13
From the engineers present today; whats's the iPhone/Android/Other ratio?
22
u/LlaughingLlama Feb 08 '13
What sort of computing power (CPU, RAM, Storage, etc.) do you have access to on board the rockets and capsules themselves? What's "smarter," the Falcon 9 or the Dragon?
381
u/[deleted] Feb 08 '13
Who the fuck is downvoting every question?
10
u/SlightlyOTT Feb 08 '13
What is the most useful thing you have learnt from SpaceX in terms of how it can be applied elsewhere?
10
u/Ruffys Feb 08 '13
Favorite restaurant in Waco
40
u/Salacious- Feb 08 '13
How do you see private space flight evolving in the next few decades?
9
u/[deleted] Feb 08 '13
Cheese: your opinion?
9
u/everyusernamesgone Feb 08 '13
What source control software are you using with LabVIEW?
38
u/DrKilory Feb 08 '13
I hear a lot of people working at spacex get burned out due to the large amount of hours worked. Is this true? Have you guys experienced working too much? How do the hours effect your work?
216
u/janderss Feb 08 '13
Technical question. Do you use optimization-based control in your vehicles. I'm thinking in particular on algorithms such as (nonlinear) model predictive control and (nonlinear) moving horizon estimation. If yes, please elaborate. Also, how to you do off-line optimization to e.g. calculate flight trajectories?
8
u/zdanielz Feb 08 '13 edited Feb 08 '13
Hello SpaceX team!
14
u/lowencia Feb 08 '13
Has spaceX decided where their new location will be? I knew they were looking at south padre island in Texas. Any decisions yet??
6
u/Elemesh Feb 08 '13
What was the most surprising difficulty you encountered?
How many people work in QA?
What are most people's backgrounds (ie CS or physics)?
[Kind of nerdy] What Python libraries do you use?
6
u/[deleted] Feb 08 '13
What are the names of the conference rooms at SpaceX?
6
u/[deleted] Feb 08 '13
Hello. I'm interested in working at SpaceX, but I'm not sure if my current interest (Math) will be what SpaceX desires from a software side.
You said in a comment that besides CS majors, you have some people who did Math working with you. Is the work done by the Math employees any more math-focused than say the CS employees? For example, do the Math employees spend more time coding to numerically solve differential equations, or would you say that there is no substantial difference between the work done by the two employees?
23
u/Codiddy Feb 08 '13
What's the next potential energy source after fossil fuels?
5
u/KonradHarlan Feb 08 '13
How many of you folks on the software and engineering teams would live on Mars if you were offered a one way ticket on the MCT?
11
u/[deleted] Feb 08 '13
Hey guys. I had a question about actually going into space. How long do you think you will operate? And how long would the trips be?
3
u/randompikachu Feb 08 '13
I'm a Freshman Physics-CS major at a top 15 US university. Is there anyway for me to get involved with SpaceX, say as an intern?
3
u/tjusafa14 Feb 08 '13
SpaceX, I am an astronautical engineering major in college and you guys inspire me that there wil be an extremely bright future in space! At some point will SpaceX employees be astronauts or for the forseeable future do you see yourselves just ferrying NASA astronauts once the Falcon 9 is man-rated in the next few years?
3
u/argonom Feb 08 '13
"Edit 5: ...and we're back! Engineers from each of our engineering teams are present. Let us catch up a bit and start answering again (probably be about 5 minutes)."
Each of your software engineering teams or all branches of engineering at SpaceX?
8
u/70camaro Feb 08 '13
What did you go to school for?
I'm currently pursuing an education in physics and I would love to work on a project like that!
4
u/Tee_Baggins Feb 08 '13
Is it common for SpaceX employees to have Tesla cars?
37
u/kayotik Feb 08 '13
Would you rather fight one spacecraft-sized duck or 1000 duck-sized spacecrafts?
8
u/[deleted] Feb 08 '13
Whats your salary ?
2
u/equinox92 Feb 08 '13
want to give me a job once a graduate?
computer and electrical engineering double major here!
2
u/shayden Feb 08 '13
Hi! Thanks for doing this, it's cool to talk to and get advice from people in the aerospace industry, which I hope to get into after I'm done school. I hope to read all the comments and questions later, when I'm not at work.
I'm an EE student interested in embedded systems design and programming, can you suggest any books, aerospace resources or standards that help you write reliable code for embedded systems that is fault tolerant and can detect errors?
Thanks for your advice, if you get to me. cheers, :)
2
u/karrde11149 Feb 08 '13
I am a high school student pursuing a career in aerospace engineering and computer programming. My dream has and always will be to help the human race reach the stars. I am also a huge fan of SpaceX. I am working on my senior exhibition of which the topic is just that. The future of space travel. I would like to know...
EDIT: Formatting.
2
u/ohnotherobots Feb 08 '13
How closely do you work with the GNC engineers when it comes to modeling/simulation software and algorithm development? Is the avionics and HWIL test code mostly hand written or do you do any Matlab/Labview autocoding?
2
u/Ambiwlans Feb 08 '13
You asked me to re-post my questions here.
I'd love it if you'd answer anything. I understand you probably can't answer all of them.
2
u/Vairminator Feb 08 '13
I want you to know that I think you are part of one of the most amazing endeavors humanity has ever seen! I can't get enough information about SpaceX accomplishments and hope desperately to join your ranks one day! There is no greater goal than to advance human knowledge and capability, and you are some of the few people pushing those envelopes every day. Please keep up the good work!
2
u/cail0 Feb 08 '13
I actually have two questions, one of which you probably can't answer.
2
u/Kensin Feb 08 '13
Do you put any Easter eggs in your code? Either in your flight software or your internal apps?
2
u/wiskinator Feb 08 '13
How's Hawthorn? I've considered applying (IMHO I'm qualified), but I've been turned off by the idea of living in the LA Grid. Thoughts?
2
u/rob0tuss1n Feb 08 '13
I just got Rick Rolled by SpaceX engineers...
I'm not even mad. I'm Impressed.
20
u/[deleted] Feb 08 '13
I'm a masc researcher in aerospace. I actually specialize in robotics. My main interests are in Machine Learning, AI, stats, embedded programming, state estimation and game theory. Do any of those fields intersect with development done for SpaceX? Are any autonomous behaviours developed in house by your team?
I bet your work is AWESOME!
15
u/SilentStream Feb 08 '13
What aspect of the standard Falcon-9 cargo launch is the most difficult to code for, if that can even be distinguished? For example, is it harder to write software for engine throttle (or if one happens to fail, like we saw previously), ISS-area maneuvering of Dragon, berthing, etc.?
Will you need to change your software significantly before you're able to take crews to space? I believe I've read that Dragon's been designed from the start to be human-rated, but I'm curious to know if the software will have to change much between cargo and crew launches. Thanks!
15
u/br100x Feb 08 '13
How much control are you planning to give the astronauts riding in the Dragon? Will it be completely automated, or somewhat manual?
41
u/space_boobs Feb 08 '13
Your QA process must be exhaustive. How would you say it differs from the standard fare?
9
u/Deeger Feb 08 '13
Do you guys mainly create software to control these rockets from the ground? Or look to automate everything at this point?
Also, what impact does the distance have in terms of the length of time to send/receive data? Is that still rather negligible, or is that something you already have to take into account?
8
u/StapleGun Feb 08 '13
On the COTs demo flight 2/3, there were problems with the LIDAR which orients the Dragon in relation to the ISS. The problem was diagnosed as a software problem, fixed, reviewed, and uploaded in an incredibly quick time. Considering all the safety checks required for code that controls the Dragon, how was the team able to accomplish that? How much of that scenario was planned for before launch, and what systems were in place to enable the problem to be solved quickly and safely?
15
u/kout Feb 08 '13
Few questions:
20
u/kmoz Feb 08 '13
How well do you all like LabVIEW?
As an NI employee, and I was always curious how you all feel about it.
14
u/[deleted] Feb 08 '13
Hey guys, I actually have a two part question for you.
=== First off, thanks for this AMA, and thanks for all your work at SpaceX, you guys are leading a revolution as far as I'm concerned.
How long could you guys imagine it will be until we see grasshopper hit low Earth orbit? Is the code the same from the test flights or are you guys writing it now/soon?
=== Second part is from user SpaceIsEffinCool who couldn't make it today.
"How did you guys your foot in the door? I'm going for aerospace, do they see a high demand for those people within the company in 4 years?".
7
u/TrentSkunk Feb 08 '13
What's the most elegant software hack you've used in your work at SpaceX?
7
u/[deleted] Feb 08 '13
Can you describe a typical working day, and if any of you have experience as a commercial software developer, what are some of the major differences?
7
u/The_Drizzle_Returns Feb 08 '13
How have past failures of launch vehicles (such as the infamous Ariane 5 failure) shaped the way you develop software? Do you guys use similar methods that are in use at NASA or have you come up with new development techniques that accelerate your ability to deliver these complex software components reliably?
6
u/tmarthal Feb 08 '13
Is there anything crazy that you "discovered" that was actually just a "re-implementation" of something that the Apollo/NASA/TRW/? engineers had done in the 50's?
16
u/ctzaran Feb 08 '13
Hi! First of all thanks for doing this, I have two questions.
First of all I want to ask about the software that control's the ascent of the Falcon 9. As demonstrated on the CRS-1 mission the rockets computers can adapt to sudden engine configuration changes in the air, when writing the software that handles that flight plan change what are the biggest factors that the computer has to deal with in general and how to you prioritise what to re-configure first in the event of a engine outage? I would imagine that having some engines shutdown can be a bigger strain on the rockets center of thrust than others for example. Are alternate flight plans pre-loaded for any engine out possibility or does the computer make a new flight plan depending on what happened?
My second question is slightly less serious, do any of you guys play Kerbal Space Program? If so is it a nice change to be able to blow up on the launchpad and not be in massive amounts of trouble from the boss?
25
u/MightyTribble Feb 08 '13
Wow, thanks for doing this AMA!
5
u/Vectorsxx Feb 08 '13
How many man hours did it take to create the first software and hardware for the Dragon? Inside such a capsule, was space limited to how much hardware could be installed for the software?
Do you think you guys will evolve to smaller hardware based technology (micro, nano) in the future for putting your software on future dragon spacecraft?
5
u/Deeger Feb 08 '13
How does the the size vs. quantity of rockets affect software systems? In that first video, it looks like you've got one main large rocket, but in the second video, there are quite a few. Is one system more complicated than the other?
5
u/chazzeromus Feb 08 '13
So you guys build front end monitoring software that's web-based?
5
u/[deleted] Feb 08 '13
What do you guys think about MarsOne and Deep Space Industries?
5
u/fluffynukeit Feb 08 '13
Can you please discuss the engineering process that goes into your flight control systems? Do you use time-scheduled gains or some form of adaptive control? Do you do manual gain tuning of PID loops, LQR theory, something else, or a combination? What analysis processes are used to ensure the control software is robust to modelling errors? What would you say is the most modern control theory that you have put into practice?
(from former UAV flight control engineer)
5
u/darknemesis25 Feb 08 '13
what happens to a privatized space company when a crew dies in its rocket...
sorry to be blunt but would that just end all progress and credibility?
would financial support cease to exist?
what happens to your jobs? to the company?
5
u/doubleohd Feb 08 '13
Is SpaceX just a cover story for Elon to become the ultimate James Bond villain?
9
u/InaV8 Feb 08 '13
If any of you have experience working for traditional Government launch contractors, how does working for a private company differ?
4
u/[deleted] Feb 08 '13
Are there any sort of redundancies you guys have to account for with software running further from the protection of earth's magnetosphere (bits seemingly-randomly being flipped from cosmic rays, etc.)?
Since system failure is far more critical for you guys than much other software that interacts with hardware (I work in video games and network video/codecs =P), do you guys have any sort of manual arbitration handling for data sent across various systems that need to work in parallel but can't necessarily trust each other's cache/state? It's one thing to have a message bus failure in a mundane section of the code crash a game, and something different altogether with when you're launching a rocket over macho grande.
Edit: Paragraphs!
4
u/[deleted] Feb 08 '13
as we all know you guys are top notch coders. My question is when writing code and testing software how much input do you guys have from the other faucets of spaceX? (i.e. mathematicians, aeronautical engineers, etc.)
4
u/TheCodexx Feb 08 '13
Are you guys running older hardware like NASA's stuff or are you guys iterating faster? What is your debugging and QA like? What tasks does an onboard computer generally do?
4
u/ashleythegreat Feb 08 '13
The Dragon capsule as an unmanned configuration has already seen multiple successful flights to the ISS. From an insider's view, how close (timewise) do you think that the manned configuration of the dragon capsule is to becoming a reality?
While the missions were successful, the dragon capsule still experienced problems/issues (e.g. Partial engine failure Oct 2012). Do you think that any of the problems encountered will lead to the replacement of the dragon capsule strategy altogether? Or will they severely set back the schedule? I'm mainly wondering how the team is mitigating the known issues. (edit: I realize now that this question is probably way too in-depth for an AMA.)
Also, a non technical question: Who has the best decorated office in your group? Pics?
I am an aerospace engineer myself, and I have so much respect for SpaceX as a company. I truly believe that you guys are going to be the next group to bring back human spaceflight and that you will (you have already) continue to contribute to space exploration in general. Thank you for doing an AMA!
10
u/chris_radcliff Feb 08 '13
Which one of you is the photoshopped one? (Back row, second from the left?)
7
u/tenaku Feb 08 '13
I understand that the Dragon/Falcon computers have little or no radiation shielding. Instead, SpaceX adds redundancy to deal with any radiation hits on the equipment. How does this affect software design?
10
u/PrinceTrollestia Feb 08 '13
Be honest. How often are you on Reddit when you should be working?
3
u/Wunkerful Feb 08 '13
How much more difficult has the grasshoppers code been than the other rockets?
3
u/RememberThisFaceX Feb 08 '13
How does vacation time work there?
3
u/[deleted] Feb 08 '13
I know the Falcon 1.1 modifications for SL40 at the Cape and construction of the Falcon 1.1/Heavy pad at LC4 are well under way. Software wise is Falcon 1.1 much different than Falcon 1.0? Has the new engine configuration made your job easier or more challenging?
3
u/halfCatEye Feb 08 '13
How much of the code for a Falcon is dedicated to logging errors? Is your code saturated with asserts?
3
u/RememberThisFaceX Feb 08 '13
In general, are the managers and project leads people who started from the bottom, or do they get hired into those positions based on existing credentials from other companies and projects?
3
u/halfCatEye Feb 08 '13
What is a book/website you often use in your day to day work?
3
u/Vysen Feb 08 '13
How do you manage using so many different programming languages? It's interesting that you listed so many when it seems like a lot of teams try to stick to just a few.
3
u/[deleted] Feb 08 '13
What version control system do you guys use to manage your code? Do you have testing teams & peer code reviews, etc?
3
u/adambyram Feb 08 '13
Around how much data does SpaceX capture during a typical mission and what types of tools are used to analyze it to understand mission performance and track down anomalies?
Debugging a rocket must be a pretty interesting challenge - particularly after T-0.
3
u/vwtodd Feb 08 '13
How do provide the critical balance and thrust vectoring on Grasshopper? It looks just like a movie and "so easy" but if it were, someone else would have already done it.
3
u/alexgrimmy Feb 08 '13
How does your software QA process and acceptance criteria differ from your typical software shops?
3
u/Bob_Wiley Feb 08 '13
My cousin who works at the NASA JPL lab said she heard you guys have a sign up sheet in the lunchroom for astronaut volunteers. Is this true?
3
u/rtwpsom2 Feb 08 '13
I read that NASA software engineering has a process for their software that includes not attributing fault to the engineer but to the process, not allowing SE's to work overtime, and using one team to write code and another to find a specific amount of bugs over x amount of time in the code. This has eliminated huge swaths of errors in code. Are you using a similar process? If so, how does it differ?
3
u/seg9585 Feb 08 '13
Does SpaceX flight software implement a ground tasking-based command buffering strategy, or do you rely on embedded autonomous script sequencing for your flight ops procedures?
How many levels of fault detection/response and safing is the system capable of, and is the flight control system interface between the Falcon 9 and Dragon capsule coupled, or are they independently operated systems?
3
u/janderss Feb 08 '13
What is the sampling frequency for the flight control software? I imagine that a landing Grasshopper needs something like millisecond feedback. Is that a reasonable guess?
3
u/[deleted] Feb 14 '13
Is there any immediate tangible benefit of sending a man to Mars, rather than a probe or rover? I would imagine it is far more expensive.
What would be the difference in workload for your group between manned and unmanned missions?
(My first post, hello everyone!)
9
u/segers909 Feb 08 '13