Project build speed importance
Filed under: Configuration Management, Development, Linux
I began writing this several years ago but never published it for various reasons. I think some of the thoughts are still really interesting, I joined a company named Pelagicore some months ago and we do some rather large builds at this company, similar in build times in the project I refer to in this text, but with some major differences. This build is based on yocto and it actually works really well once you get through the first build (currently a scratch dev image build is up to 4 hours in time, but after that it’s able to rebuild only changed software and recipes as necessary. The scratch build is heavy admittedly, but we are working on some ways to improve that as well, such as sstate caches, icecc distributed build systems, and so forth. I think with modifications we could get those 4 hours down to under an hour, which would be really sweet. Anyways, I thought I’d post this text as is, even though it’s not finished, just because and since I re-read it right now 😉 .
Original text
Again, I’m struck by the importance of a proper build process in projects. No matter how small or big the project is, the build must be kept fast and working from get go to the end — which means its end of life cycle. I am currently working in a large project with a huge code base where the build system is, in my opinion, next to completely collapsed. Build time is in excess of 1-4 hours, and the automated test-suites take from 3 hours to 7 hours to complete, depending on how thorough a test-suite you choose. A gigantic heap of time is spent just…. waiting… waiting…. waiting. Just to make a simple one line edit and then compile to see if it compiles can take up to 4 hours. I’ve previously been a bit spoiled with good and simple build systems, only occasionally running into really crappy build systems — for some reason, a lot of scientific open source software seem to be stuck in this category.
This time, I think I hit pay-dirt on “how not to do it”. Instead of focusing on the bad parts, I will try to focus on the things to do and to keep in mind. Nobody seems to like a grumpy person anyways, which I really am sometimes.
1. Keep the build system simple and manageable. Try to maintain the build system in a logical fashion and in a single language/system (scons/python, Makefiles, etc).
2. Expandable (new directories, files, etc)
3. Scalable (multiple CPU’s, machines, threads, etc)
4. Try to use as few frontends as possible (a single top makefile for example, with targets depending on what you want to do), and keep people informed of changes. Wiki with history via RSS is _perfect_ for this. This doesn’t mean you can create a script to build something, then 2 months down the road delete it because it is no longer needed, it causes a lot of stress for developers who just barely had a chance to find the script.
5. The programmers are (most likely) your customers.
DIY OSD update
Filed under: Development, FPV, Hardware, Robots, Video
Late update, but video is working as expected, GPS is working to some extent, but because of local weather I havent really tested it outdoors… I’m considering changing video transmitter and power system however. I’m currently running on 1S 160mAh batteries with a stepup regulator to 5v to a 10mw transmitter, which gives a really lousy battery time for the video system, also I’ve got too many batteries flying around at the moment I think.. Limiting to a bit fewer might be a good idea. Currently, there’s a battery for the video link/osd, one built into the camera, one for motor/rc receiver/servos, and one in the rc transmitter, 4 batteries in total of various sizes etc… a pain to charge them all. Hooking up the videolink and OSD to ESC/BEC is a trivial change and would remove a lot of cruft (stepup module etc), but will also add the EMC from motor/ESC to handle for the videolink etc as it’s a completely separate system today.
I’ve also upgraded so I’m using a mobius actioncam. Everything all in all means the system will draw more power and weigh more as well. Removing some batteries and running from the mains system would be really nice.
qmake project custom install and files being installed as non-executable
I had some issues with creating a custom install target from a qmake pro file for a project today and everything took me quite some time to get working I’m embarassed to say. This was done to create make install_ptest target for a yocto build, but the files kept turning up non-executable in the rpm’s and install directory. In the end, I got it working by adding the executable keyword to the CONFIG of the target, in addition to the no_check_exist.
PTEST_FILES += \
$$ROOT_SOURCE_DIR/tests/service_tests/test_runner.py \
$$ROOT_BUILD_DIR/tests/service_tests/tst_service_tests
ptest.depends += $(first) sub-tests tests
ptest.path = /usr/lib/AnyApp1/ptest
ptest.files = $$PTEST_FILES
ptest.CONFIG = no_check_exist executable
INSTALLS += ptest
QMAKE_EXTRA_TARGET += ptest
I hope this helps you if you run into the same problem as I had some issues finding information about that specific flag or the problem.