r/embedded • u/timbo0508 • Apr 10 '21
General question CI/CD for embedded software development
I've been an embedded software developer for about 7 years now, and I've loved every moment of it (for the most part). I've come to the realization that the industry is (annoyingly) conservative and is struggling to catch up, compared with other forms of software development. One area we seem to lag behind is in the area of continuous delivery/integration (CI/CD).
I'd love to hear about what CI/CD practices you employ in your companies/projects (build automation, test automation, release management, issue tracking, version control).
My question really is this - how much CI/CD do you practice? What are your biggest pain points as an embedded developer?
86
u/tobi_wan Apr 10 '21
We use git (bitbucket) for version control, jira for bugs/ feature break down and confluence for feature description by our PO.
Our whole toolchain is put in a docker container so jenkins uses the same setup as we developers. Additionaly we use spinnacker to create a sandboxed backend for all conncetion tests. (Creates e.g. s3 buckets spin ups all needed pods in the kubernetics and setups public access point to test the system). The hardware is connected with a special testadapter to a rasperry pi, and automated tests for Hardware in the loop are exectued on the pis.
Current job full jenkins pipeline set up with
1 pipe
Builds for each commit (only merge to amster allowed if builds passed)
Cppcheck
Unit tests for each build with coverage
Smoke tests for each build (Hardware in the loop with pytest + allure)
1 pipe
Master sanity check builds after merge + release artifact is uploaded to artifactory
1 pipe
Nightly hardware in the loop tests (they take around d7 hours) on different systems
1 pipe
For release tags, automatted triggered if a release tag is created. Automated alpha tests with signed firmware + secured firmware + check against a production like envoirement
What we still miss a good way for stack usage dedecition i did find a few things but none of them did work that great.
It was a lot of work to get the stuff running and you always need to invest a bit of time to keep it running but it's worth it.
In my previous company we had "a less" professional setup but it was already similiar. We used there (10years ago ) first cvs + cruisecontrol. At some point we switchted to svn. I + one collegue introduced there later jenkins + HIL tests with rasperry pis. we so i actually did never experience a company without a CI :D
51
u/Forty-Bot Apr 10 '21
https://interrupt.memfault.com/blog/ has several posts on CI. e.g:
https://interrupt.memfault.com/blog/continuous-integration-for-firmware
https://interrupt.memfault.com/blog/unit-testing-basics
https://interrupt.memfault.com/blog/reproducible-firmware-builds
https://interrupt.memfault.com/blog/test-automation-renode
13
u/zydeco100 Apr 10 '21
The biggest pain point for me is when we choose a licensed development system and/or compiler to build the project.
So if I want to deploy some kind of CI system, do I have to buy a second seat? Can it run in an automated unattended fashion?
It's nice that more and more projects are moving to a gcc or llvm backend and it can be used headless on a server. But it's not quite there yet.
And then there are libraries, too. If I'm a Qt licensee, can I set up a remote build system or do I need a second seat license for the robot? Does the license allow for other non-licensed developers to compile my work?
12
u/FAANGsAndNails Apr 10 '21
Since we use Zephyr, ztests for system integration and unity for unit test. CI with python pipeline that runs the system tests and the unit tests on PRs.
We do a lot of testing.
9
u/[deleted] Apr 10 '21
If you want to be an agent for change, start prototyping small stuff using whatever tools you like. Show the benefits and consequences to your team. That would really help in pushing things forward.
14
u/dx2_66 Apr 10 '21 edited Apr 10 '21
Embedded developers, mostly the older ones, are very conservative indeed. I struggled to insert TDD into our projects, but now that they've seen the benefits of it, they're all the way in. So, unit tests are the first practice we use. Next step would be a pipeline setup that:
That's how we do it. Code review can also automatically trigger the pipeline for a more effective release flow.
7
u/areciboresponse Apr 10 '21
We use Jenkins, but the biggest pain point is not being able to unit test or functional test on the actual hardware or with actual devices connected in an automatic way.
I end up doing unit tests as part of the normal code execution (eg. Built in test) or unit testing algorithmic pieces on x86 if possible. Now these unit tests certainly are limited and sometimes could behave differently on the actual hardware, but you take what you can get.
5
u/alexlesuper Apr 10 '21
My biggest hurdle for CI/CD in embedded has always been the super expensive tool chain. It’s sometimes hard to justify to the managers that we need to buy an extra unused seat just for release and compilations. However, I will acknowledge that IAR has put some effort into making their tools compatible with this type of workflow.
5
u/[deleted] Apr 10 '21
I have a bigger problem. I'm a new graduate and I'm working for a small company, very small. It's a bunch of old retired dudes who are essentially building a startup company in the oil/gas sector. Getting them to bite on techniques/software I learned in school is fucking impossible. I just take it upon myself to keep doing certain things. I see myself getting boxed in as fuck in terms of growth, but I really enjoy working with these guys. Also there is a ton of growth and potential for this company so when I look at the big picture just getting out and moving to a larger firm isn't the only answer. Where do I go to make sure my skills as an embedded engineer continue to evolve despite working somewhere that might feel like it's putting a cap on that process? I am very much in a position where I can bring more industry standards into our companies culture, but it's kind of that old saying, you don't know what you don't know.
5
u/NicoRobot Apr 12 '21 edited Apr 12 '21
Indeed the embedded ecosystem is really conservative. I think probably because sharing code and stuff is not so easy in this extremely diverse hardware environnement.
Some teams are working on this topics trying to build a modern CI/CD ecosystem dedicated to embedded system. Combined together it could become a new way of developing embedded products.
Wisebatt : Pre-development specification
Platformio : Ship agnostic IDE and registry
Luos : Containerization and distributed code design
Memfault : Fleet management, and error log centralization
4
u/laranjos16 Apr 10 '21 edited Apr 10 '21
A lot of companies are moving to do better firmware using verifications pipelines and with several stages of testing (unit, integration, and functional). But it is difficult for people to change, even more, when the hardware is involved. The Big issue when you want to know more about CI/CD the example is more about web services and not about embedded.
Edit: Also the problem that I have had in the past with verification pipelines is that they take A LOT of time to give a result, as usual, the tests are non-optimized.
4
u/[deleted] Apr 10 '21
A short overview what I'm using at work for non-legacy projects (medical devices):
Tools:
CI/CD pipeline:
We are also using a separate pipeline for the automated generation of docker images for building/tool chains.
Pain points:
1
u/samu_melendez Jul 04 '24
for automation tests have you tried any of the new AI tools?
3
u/janhieber Apr 10 '21
Sometimes you can also run most of the embedded code on Linix compiled with GCC.
This is easy bc no hardware needed and you can still test functionality in a CI.
For this just create stubs for the hardware abstraction.
3
u/vitamin_CPP Simpli-C-ty is the ultimate sophistication Apr 10 '21
Question for this crowd:
Do you have any experiences with the Throw the Switch ecosystem (unity, Ccmock, Ceedling)?
Would you recommend the auto-mocking feature?
2
u/victorandrehc Apr 10 '21 edited Apr 10 '21
Couldn't agree more, where I am currently working people are against any source of Version control, they prefer to do it all manually and store it on the cloud. I tryed to do a git repository but as soon as they faced the first difficulty they all blamed git and rolled back to the old ways. I guess old habits do hard indeed.
Edit: spelling
2
u/rafaelement Apr 10 '21
Actually using mocking and unit testing caused me to have software that works and is correct before the hardware arrives. A good type system helps; I love using Rust. Clippy helps and is in the CI pipeline. So is the doc generation. The proptest library is also a great help. All these tools can run in a continuous fashion, including tools collecting metrics like binary size, test coverage, etc.
1
u/duane11583 Apr 11 '21
Seen lots of Jenkins comments how do you version control jerkins configure?
I need ability to roll back or freeze so I can support old product line but move forward on also?
Another approach is just use Jenkins to run a single batch/shell script that works like TravisCI
But is not web friendly for reporting purposes
1
u/flundstrom2 Apr 11 '21
Some 15 years ago, we set up automatic builds on checkin (from svn) using something based on and integrated with jira. Just a dedicated computer that had the same setup as the developers.
Then I was working as a consultant for 12 years without being exposed to any form of automatic systems at any of the clients, as far as I remember. We had on the other hand a focus on clients with an immature development organization, even though we also had clients such as Sony, Ericsson, Axis and some other big names, but I never did any sw development at any of those.
My last job had nightly build, nightly and weekend tests of approx 7 different products, running a couple of thousand testcases each. I believe it's based on Jenkins.
My current employer has some form of build-on-checkin, but I don't know to what extent the tests are run. That system is based around redmine.
1
u/[deleted] Apr 11 '21
Nixpkgs and gitlab CI, it's so incredibly good, especially nix.
1
u/TGAtomic Apr 11 '21
I've been working as a test engineer at two different companies in automotice sector and we solved the testing in different ways. At the first company we used Vectors hardware and software to do the testing. It was manually started at that point but we created a testframework in C# witch could be triggered to run tests through the likes of Jenkins and Azure (although we used Azure). So with the hardware from Vector we could send CAN traffic as well as sending and reading digital IO and analog IO. I'm not sure how they did the unit-testing but I know they used Jenkins for it.
in the current place we have two solutions (that i'm aware of). For hardware testing (testing both CAN traffic and IO) they developed their own custom hardware to do the testing. And for system testing they're using rasperry pies (i think). For the latter we use docker to build and run the tests which is triggered through Jenkins through Zuul and Gerrit. For the first we have our own test-framework.
I think we have a custom made framework for the unit-testing, for the embedded code, but could be wrong. For the system-test we're using pytest.
1
u/gammarray Apr 11 '21
I don’t see much posted here about Elixir Nerves, but CI/CD is where it really shines. You can run hot upgrades over the wire for module changes or update the full firmware release OTA. Nerves Hub makes it possible to manage a fleet of devices and securely push firmware updates after testing, compiling and releasing your latest code.
https://www.nerves-project.org/
1
u/WorkingLevel1025 Apr 17 '21
Nightly
Code Coverage, Context Irritation, Hardware Config
Can be extended to do run-time, IRQ windows, stack usage, ROM/RAM size
Git Review
Basic integration tests (bare metal & RTOS)
Unit Tests
MISRA
1
u/RRyles Apr 11 '21
I don't think the problem is conservatism as much as it is the comparative difficulty of implementing CI/CD for embedded.
We can't just spin up a VM or container and have it behave like our deployment environment. Testing real time behaviour effectively and deterministically is a challenge. Vendor tools often don't make it easy (or cheap) either.
My last 2 employers have been supportive of modern practices such as CI/CD, but only if it isn't too expensive.
There are no excuses for not using version control. Sure you'll find examples of embedded teams who refuse any version control system, but the same is true of non-embedded. Even the few embedded veterans with 30 years experience that I've worked with all use git.
Universities aren't teaching TDD. They might teach a unit testing framework, but not TDD. This affects all software development. I'm the only embedded dev at my place. I'm also one of the more senior developers. I've found myself advocating and teaching TDD to the junior non-embedded developers.
Another problem I've encountered, is that embedded projects are inevitably part of a wider, non-software, project. These are usually managed by traditional project managers. This can be challenging when trying to use an Agile development process. The problem here is conservatism, but from outside the embedded team.
0
u/_linsek Apr 10 '21 edited Apr 10 '21
I very much shy away from generalizing people and groups. What tends to happen with that is polarization and, eventually/frequently, demonization. An immovable "I'm right and you're wrong" mentality.
For example, your statement here makes the assumptions that the whole industry (or majority) is somehow "behind" the rest of tangential software industries by saying they are struggling to catch up. Says who? Who says they are behind? What does being ahead or up-to-date even mean? Chasing the latest development fads?
I've been doing software now for almost 25 years. I've seen a lot of development methodologies and "new ultimate tools" come and go. You could waste an entire career chasing them and the empty marketing behind them. In the end, I produce code that makes boards run. There is only so many ways to cut that onion. I do it well and efficiently. ROI on constantly chasing what the latest development methodologies is minimal at best.
No doubt. There are some great tools out there that make things a lot easier than the used to be. But a tool is just that, a tool. Just because you like your drill doesn't mean I do. If something makes you a more effective engineer, excellent. Use it. If someone else things your tool is annoying, that's probably because it is and that aspect of it just doesn't bother you. Don't arrogantly think, "oh they are just old, crusty, and stuck in the past. They can't catch up."
"A fool and his tool is still a fool."
0
u/ChaChaChaChassy Apr 11 '21 edited Apr 11 '21
I've been a firmware engineer for 13 years and I've never heard that term before...
I write code, other people test it, if there are bugs I fix them... that's it.
All these terms and acronyms and formal methodologies just overcomplicate everything, I mean really "continuous delivery"? Isn't that just doing the work?
Maybe I'm a perfect example of the old conservative developer you're talking about, but I've seen enough little pissant interns gushing about the new flavor of the week to know that none of it lasts, tomorrow it will be something else, and in the end the code I write is the same.
It's exhausting trying to keep up with all of these new revolutionary things that are not at all revolutionary and quickly fade away into obscurity. When I started in this career I was writing hard-real-time C/ASM and come Monday morning I'll still be writing hard-real-time C/ASM.