Gitea and Drone.io
Filed under: Configuration Management, Development, Projects
Recently I’ve been frustrated at work while using Gitlab and Jenkins for various reasons, some of the integrations are really fragile due to some plugins we use, both Jenkins and Gitlab are incredibly bloated and use insane amounts of resources and they are simply not reasonable choices for a private setup. Also I recently replaced my server at home (basically a machine that does almost everything I want at home) from a 32 bit atom to an I5 intel machine with 16 gigs of RAM etc, which means I have totally different resources to work with. For example, 32 bit i386 cpu’s are not supported out of the box by docker, and the cpu was quite overloaded. With the new box I’ve been able to play around a little with my setup.
I’ve previously used just a basic and manual git setup at home, with approximately 110’ish repositories in it. I’ve been playing around with gitea and drone.io at home with the new server and am very pleased with it, even though it took a bit of work to get used to it. In all honesty, I’ve only done fairly basic work with it so far. The only really complex stuff I’ve done was to move my existing git repositories into the gitea environment by scripting a bit and using the gitea web based API.
Regarding gitea, so far I’ve noticed the following things:
- Very slim by comparison to other options, currently uses around 60 MB of RAM and not a lot of CPU from what I’ve seen. Especially considering what you get.
- UI is fairly similar to gitlab/github.
- Setup was very simple except for database connection (I winded up just using sqlite3 I believe, I was lazy and also don’t expect more than a very few users).
- Pull Request is really nice.
- Issue tracking seems to work fairly well.
- Docker setup with volumes is very easy.
- Seems to have the essentials in plugins etc that I need.
- API seems very nice, I’ve only used it for the migration so far though.
- The only bad part I’ve seen so far is that the administration panel might be a bit spartan at times, but I don’t really mind.
Regarding drone.io, my first impressions are:
- I absolutely love the yaml file format so far.
- UI is incredibly clean, on the verge of too clean.
- Integration with gitea was super simple once I actually got it working (documentation was not 100% accurate I think).
- Simple to get started with if you have a sane build pattern.
- Nice integration to gitea and you get marks on build statuses etc. Interesting to find if you can block a build from being merged based on build results as well.
- I’ve managed to make a simple build of a playground project I have by adding a Dockerfile which is built into an image when the build is started, then a continuation of the build builds my project inside the docker image we just built.
- First time using docker-compose so it was a bit of a hassle understanding this, but it was fun ;). Not always obvious where some configuration should be placed etc.
- Yaml file format definitely not enough for the type of pipelines we do professionally though :(.
- Pleasantly surprised you can actually add and remove build slaves to the drone.io platform.
- Also pleasantly surprised by how to do parallel build steps. Syntax is super simple.
- I really lack some form of artifact storage, or at least a plugin for something that is not either cloud based or incredibly enterprisey (artifactory). Actually, I’ve had issues just finding a good light weight open source artifact storage so far…
- I also lack some form of nice presentation of various build artifacts, code coverage or unit test results etc.
In all, pleasantly surprised by how simple this was to setup and configure. It was a fun trip and I’ll continue using it at home for now.
As a sidenote, for the stuff I have on github, I do like to use travis, it also has a nice syntax and is a nice solution.
Bugs, bugs and more bugs
Filed under: Development, Linux, Projects, Ubuntu
Lately, I’ve come to realize more and more that bug handling in open source, and specifically in Ubuntu has dramatically declined in efficiency. For years I’ve been extremely satisfied with using Linux because it’s bug free, there has simply not been any serious bugs that I’ve run into. In the last weeks, I’ve run into several more or less serious bugs in Ubuntu, which got me looking at how the bug handling is done.
First off, a few weeks ago, I ran into a bug with Ubuntu 10.10 Ubiquity (the Live CD installer) where I accidentally marked my old /home drive as ext4 when it was ext3 (but not to reformat it). The installer complied happily, and set it up as ext4, but once it got back online, the harddrive was completely wiped. No warning, no nothing. I started looking around, after a while I’ve found several reports on the same matter on launchpad. For example this and this.
This lead me to take a look at Ubiquity’s other bugs in launchpad, and it’s not very promising. The main installer of Ubuntu 10.10 has 1528 Open bugs as of writing this, of which 846 bugs are new, 35 bugs are marked High importance — and the bugs I found (dare I say, they seem Critical to me, are still not marked with any importance at all). Only 12 bugs are marked as having a patch.
Fine, maybe this is not the poster child of open source. However, the last few days I’ve been severely annoyed by the password popup which is misbehaving. I enter the password, and hit enter (or hit the Authenticate button) and the password field disappears, but the rest of the dialog stays up, and nothing works in it. The only thing you can do is to kill it with the x button. When you do this, you get authenticated…
Since I’m not sure exactly how the authentication is performed in Ubuntu for the update manager etc, I decided to check the update-manager package for Ubuntu on Launchpad. What do I see, if not another package with gigantic mass of bugs filed, but noone dealing with them. 1017 Open bugs, 520 of those are New and 15 marked as High importance. This bug I’ve been having has been reported all over the net, but noone seems to be dealing with it and it isn’t really reported in launchpad. Some computers has it, some doesn’t. It’s nowhere near a critical bug, or even a high importance one, but it’s annoying none the less and it looks extremely crude and comes off giving a fairly unstable feeling.
All this being said, I am wondering how bug handling is done, and how it should be managed on “aggregate” projects such as Debian and Ubuntu. I think the idea is really nice, having upstream bug trackers for each package in the project, but maybe we are spreading too thin having several bug trackers for each minor project? Also, how do we as “normal” users know which package is the reason for the error? I am not so sure it is really the update-manager that is the error in this case, it might as well be some completely other thing behind all that dbus stuff etc. Ie, what is the point of me filing bug reports if I’m not sure they wind up in the right place, or are at all looked after?
Unit testing and stubbing singletons
I got a bit curious about stubbing Singletons for testing during the weekend as well. We often find ourselves needing to test large codebases at work, and in the current project I’m in, we do complete end to end signal flow tests, but people are finally realizing that this will simply not do. For this reason, we’re doing a lot of work to try to split the entire project up into manageable chunks. One of the main problems has been the incessant use of singletons. A simply half-way-there to doing full out interfaces is to simply make all public function calls virtual and then create a stub class of the singleton saving the message or whatever passed on, into a variable which can be grabbed and tested from the actual unit test.
A sample of the general idea below:
#includeclass A { public: static A *instance() { std::cout << "A::instance()" << std::endl; if (!s_instance) s_instance = new A; return s_instance; }; A() { std::cout << "A::A()" << std::endl; }; // Virtual makes the difference virtual void send(int i) { std::cout << "A::send()" << std::endl; // Blah blah, send i or something }; static A *s_instance; private: }; class stub_A: public A { public: static stub_A *instance() { std::cout << "stub_A::instance()" << std::endl; if (!s_instance) { s_instance = new stub_A; A::s_instance = s_instance; } return s_instance; }; stub_A() { std::cout << "stub_A::stub_A()" << std::endl; }; void send(int i) { std::cout << "stub_A::send()" << std::endl; y = i; }; int getMessage() { return y; }; private: int y; static stub_A *s_instance; }; A *A::s_instance = 0; stub_A *stub_A::s_instance = 0; int main(int argc, char **argv) { stub_A::instance()->send(5); std::cout << "stub_A::instance()->getMessage() == " << stub_A::instance()->getMessage() << std::endl; A::instance()->send(7); std::cout << "stub_A::instance()->getMessage() == " << stub_A::instance()->getMessage() << std::endl; }
Inproductive productivity
Filed under: Configuration Management, Linux, Personal, Projects, Windows
For a while I’ve been stuck in slow speed mode again, not really doing great work, just being on average. It feels weird. Don’t really get much done, but I have on the other hand had a great deal of time to test some “new” technologies, well, new as in only 10-15 years old I guess :-). I’ll get back to that later. Also, I’ve begun a new contract at “a big company”.
This is my first time at a really giant hunk of a company, the biggest I’ve seen before was circa 500 people in all, and it moved slower (the beaurocracy) than this in all honesty. This BigCompany is quite interesting to me. Started off with almost 4 weeks of introductions, courses, and so forth. They have a dedicated TEAM of CM’s, that alone is just… wow :-P. I’ve just been put up to speed and started working a little before this weekend so I might be a bit premature, but I like it so far. The weird part is, things happen, but not as I’m used to it. I’m used to 13+hour days and frentic coding/hacking to get things to happen, everyone here eschews away with their 8 hour days — only working overtime at very special occasions — yet slowly things get done, new functionality gets added and so forth.
Another thing that kind of amazes me — and worries me to some extent — is the kind of planning that is done. I’m used to small scale projects with workpackages or task based development, where no workpackage should ever take more than 4-5 days to implement. This place uses a workpackage development structure where each package takes up to 6-7 weeks for 6-10 people to implement. We’ll see how it works out — at least their “stand-up meetings” works :-).
All that being said, I had the time to write quite a bit of python which is a first, then I’ve looked into d-bus architecture which is also a first, and I also looked into Bluetooth and how to use it — some test applications running, fetching services and graphically displaying info about all units it finds etc. The complexity of Bluetooth is rather saddening imho, it’s a horrible protocolstack to work with in some senses, even though I was really impressed by how much python does for you.
I’ve been unused to the whole concept of python before this, and just a tad sceptical. Mainly because of all the problems with version matching that you always wind up having to do, to make anything work properly (try getting scons, trac and wamp, and some more tools working on a win32 machine some day for some fun).
Anyways, I always figured there has to be an upside, and there really is — python is hackfriendly 🙂 . In less than 3-4 hours I went from writing my first simple helloworld to having a scratch written class based graphical (tkinter) interface implementing some very fundamental bluetooth commands. In my world, thats not bad at all ;).
I’ve also had time to learn a lot of new tools at work. I’ll comment on those some other day as I havent seen much other comments on some of them (some is imho very expensive crap with a nice wrappings, while some are completely awesome). Sidenote, I simply adore the systems we are working on 4 xeon with 4cores and 64 gig ram.
I’ll get back later :-).
“New” subversion structure using svn:externals
Filed under: Configuration Management, Development, Projects
Me and the boss deviced a new structure for the project during the last few weeks, and it’s been slowly refining in our heads until yesterday when we finally implemented it. I think we made a rather refined and complex structure, but once we got it into place physically and once we get the general idea into the developers heads (including me), I think it will prove very powerful.
That being said, I don’t think this is a new structure, I just think people are very quiet about how they use subversion, and it’s a problem. Newcomers do the same old errors over and over again. So, let’s get on to try and explain it all.
Most projects uses a single BTT root, where BTT stands for Branches, Tags and Trunk. Ie, they start a project, and then straight in the root put the BTT, and then inside that, they create the project structure. For example:
- project-root/
- Branches
- Tags
- Trunk
- admin
- src
- out
- test
This is a good basic structure for very small projects, containing perhaps 10’ish files, or where the actual implementation is perfectly homogenous and has no need for separated versioning. Every time we want to make a release, we cheap copy the content to Tags as a new tag (called perhaps /Tags/Milestone1-RC1). We now have a release that we can provide to people.
The problem comes if it isn’t so homogenous. For example, let’s say you are developing a calculator. It has two objects, a numpad and a display. What if you want to make a new version just of the display? You need to make a completely new version, including for the numpad.
Or how about wanting to branch just a small part of the project? Ie, I want to use a branch for the numpad, and then use the trunk for the display. You’d then have to make a cheap copy for the entire tree. Admittedly, it isn’t costing too much.
Our “new” structure deals with this on a different level. Basically, the idea is to have multiple BTT roots, and then use svn:externals to connect the correct tags to create
1) a complete releasable project and
2) a complete workarea project.
For the calculator example, you get the following structure:
- calculator/
- Calculator_Modules/
- Display/
- Branches/
- Tags/
- Trunk/
- Numpad/
- Branches/
- Tags/
- Trunk/
- Calculator/
- Branches/
- Tags/
- Trunk/
As you can see, it looks much more complex, and it is, but the possibilities are infinitely much better.
The Calculator/Trunk/ directory contains a svn:externals property linking in the Calculator_Modules/Display/Trunk as Display and Calculator_Modules/Numpad/Trunk as Numpad. This works by linking external resources into the current directory structure, so basically I would get the trunks into my Calculator trunk, but properly renamed, without them actually being there in the repository. This also works on “real externals” by the way, such as linking in a specific version of a library from some repository on the Internet.
To create a Calculator/Tags/MS1 we could either just set a -rXX to the correct subversion revision, or we would create svn:externals to the correct Display and Numpad Tags, not their trunk. This way, we can say that “Calculator 1.0 contains Display 2.0 and Numpad 2.1”, not “Calculator 1.0 contains Display revision 439 and Numpad revision 587”, or even worse “Calculator 1.0 is revision 587” which completely lacks granularity.
I’m not completely sure it’s perfect, and others have probably already tested it, but I think it will be pretty sweet :-).