FMEA of Door and conclusions

Bugs, bugs and more bugs

Lately, I’ve come to realize more and more that bug handling in open source, and specifically in Ubuntu has dramatically declined in efficiency. For years I’ve been extremely satisfied with using Linux because it’s bug free, there has simply not been any serious bugs that I’ve run into. In the last weeks, I’ve run into several more or less serious bugs in Ubuntu, which got me looking at how the bug handling is done.

First off, a few weeks ago, I ran into a bug with Ubuntu 10.10 Ubiquity (the Live CD installer) where I accidentally marked my old /home drive as ext4 when it was ext3 (but not to reformat it). The installer complied happily, and set it up as ext4, but once it got back online, the harddrive was completely wiped. No warning, no nothing. I started looking around, after a while I’ve found several reports on the same matter on launchpad.  For example this and this.

This lead me to take a look at Ubiquity’s other bugs in launchpad, and it’s not very promising. The main installer of Ubuntu 10.10 has 1528 Open bugs as of writing this, of which 846 bugs are new, 35 bugs are marked High importance — and the bugs I found (dare I say, they seem Critical to me, are still not marked with any importance at all). Only 12 bugs are marked as having a patch.

Fine, maybe this is not the poster child of open source. However, the last few days I’ve been severely annoyed by the password popup which is misbehaving. I enter the password, and hit enter (or hit the Authenticate button) and the password field disappears, but the rest of the dialog stays up, and nothing works in it. The only thing you can do is to kill it with the x button. When you do this, you get authenticated…

Since I’m not sure exactly how the authentication is performed in Ubuntu for the update manager etc, I decided to check the update-manager package for Ubuntu on Launchpad. What do I see, if not another package with gigantic mass of bugs filed, but noone dealing with them. 1017 Open bugs, 520 of those are New and 15 marked as High importance. This bug I’ve been having has been reported all over the net, but noone seems to be dealing with it and it isn’t really reported in launchpad. Some computers has it, some doesn’t. It’s nowhere near a critical bug, or even a high importance one, but it’s annoying none the less and it looks extremely crude and comes off giving a fairly unstable feeling.

All this being said, I am wondering how bug handling is done, and how it should be managed on “aggregate” projects such as Debian and Ubuntu. I think the idea is really nice, having upstream bug trackers for each package in the project, but maybe we are spreading too thin having several bug trackers for each minor project? Also, how do we as “normal” users know which package is the reason for the error? I am not so sure it is really the update-manager that is the error in this case, it might as well be some completely other thing behind all that dbus stuff etc. Ie, what is the point of me filing bug reports if I’m not sure they wind up in the right place, or are at all looked after?

Unit testing and stubbing singletons

I got a bit curious about stubbing Singletons for testing during the weekend as well. We often find ourselves needing to test large codebases at work, and in the current project I’m in, we do complete end to end signal flow tests, but people are finally realizing that this will simply not do. For this reason, we’re doing a lot of work to try to split the entire project up into manageable chunks. One of the main problems has been the incessant use of singletons. A simply half-way-there to doing full out interfaces is to simply make all public function calls virtual and then create a stub class of the singleton saving the message or whatever passed on, into a variable which can be grabbed and tested from the actual unit test.

A sample of the general idea below:

#include 

class A
{
public:
    static A *instance()
    {
        std::cout << "A::instance()" << std::endl;
        if (!s_instance)
            s_instance = new A;
        return s_instance;
    };

    A()
    {
        std::cout << "A::A()" << std::endl;
    };
    // Virtual makes the difference
    virtual void send(int i)
    {
        std::cout << "A::send()" << std::endl;
        // Blah blah, send i or something
    };
    static A *s_instance;
private:
};

class stub_A: public A
{
public:
    static stub_A *instance()
    {
        std::cout << "stub_A::instance()" << std::endl;
        if (!s_instance)
        {
            s_instance = new stub_A;
            A::s_instance = s_instance;
        }
        return s_instance;
    };

    stub_A()
    {
        std::cout << "stub_A::stub_A()" << std::endl;
    };

    void send(int i)
    {
        std::cout << "stub_A::send()" << std::endl;
        y = i;
    };

    int getMessage()
    {
        return y;
    };
private:
    int y;
    static stub_A *s_instance;
};

A *A::s_instance = 0;
stub_A *stub_A::s_instance = 0;


int main(int argc, char **argv)
{
    stub_A::instance()->send(5);
    std::cout << "stub_A::instance()->getMessage() == " << 
        stub_A::instance()->getMessage() << std::endl;

    A::instance()->send(7);
    std::cout << "stub_A::instance()->getMessage() == " << 
        stub_A::instance()->getMessage() << std::endl;
}

Inproductive productivity

For a while I’ve been stuck in slow speed mode again, not really doing great work, just being on average. It feels weird. Don’t really get much done, but I have on the other hand had a great deal of time to test some “new” technologies, well, new as in only 10-15 years old I guess . I’ll get back to that later. Also, I’ve begun a new contract at “a big company”.

This is my first time at a really giant hunk of a company, the biggest I’ve seen before was circa 500 people in all, and it moved slower (the beaurocracy) than this in all honesty. This BigCompany is quite interesting to me. Started off with almost 4 weeks of introductions, courses, and so forth. They have a dedicated TEAM of CM’s, that alone is just… wow . I’ve just been put up to speed and started working a little before this weekend so I might be a bit premature, but I like it so far. The weird part is, things happen, but not as I’m used to it. I’m used to 13+hour days and frentic coding/hacking to get things to happen, everyone here eschews away with their 8 hour days — only working overtime at very special occasions — yet slowly things get done, new functionality gets added and so forth.

Another thing that kind of amazes me — and worries me to some extent — is the kind of planning that is done. I’m used to small scale projects with workpackages or task based development, where no workpackage should ever take more than 4-5 days to implement. This place uses a workpackage development structure where each package takes up to 6-7 weeks for 6-10 people to implement. We’ll see how it works out — at least their “stand-up meetings” works .

All that being said, I had the time to write quite a bit of python which is a first, then I’ve looked into d-bus architecture which is also a first, and I also looked into Bluetooth and how to use it — some test applications running, fetching services and graphically displaying info about all units it finds etc. The complexity of Bluetooth is rather saddening imho, it’s a horrible protocolstack to work with in some senses, even though I was really impressed by how much python does for you.

I’ve been unused to the whole concept of python before this, and just a tad sceptical. Mainly because of all the problems with version matching that you always wind up having to do, to make anything work properly (try getting scons, trac and wamp, and some more tools working on a win32 machine some day for some fun).

Anyways, I always figured there has to be an upside, and there really is — python is hackfriendly . In less than 3-4 hours I went from writing my first simple helloworld to having a scratch written class based graphical (tkinter) interface implementing some very fundamental bluetooth commands. In my world, thats not bad at all .

I’ve also had time to learn a lot of new tools at work. I’ll comment on those some other day as I havent seen much other comments on some of them (some is imho very expensive crap with a nice wrappings, while some are completely awesome). Sidenote, I simply adore the systems we are working on 4 xeon with 4cores and 64 gig ram.

I’ll get back later .

Work? What work?

So, just a brief update. I’ve recently (a few months back *cough*) taken over our Linux “education” group at work, and it’s interesting. The sad part is, we mostly only see people who already knows what Linux is as we’re working internally in a world where most people are rather Computer savvy as it is. It’s given me a few new viewing angles though, and I’ll get back to that at a later point.

Currently working on some Trac guidelines for our Change Management process as well. Working from home today to actually get something done with it, as most of the days I wind up getting too many disturbing calls, talks and discussions to be very efficient. Our first two tries at making a decent workflow winded up a bit messy, and I think we really must get this down properly this time.

There are some other things I react on, and want to fix, for example, as it looks now, every single project sets up their own bugtracking/ticketsystem, and every project uses a different system (trac, mantis, clearcase, dimensions, etc). Preferably, this should be centralized in some fashion, and if possible I’d love to get a bit more homogenized environment. As it is, I try to tell people “look, here’s a system for handling your day to day tasks, use it!”. First time, the workflow got overly complex, second shot was also overly complex, and people where put off by all the choices and steps to take. This problem mainly stems from project/change management criterion.

My latest and greatest (yeah yeah) workflow should alleviate some of these problems by making some of the choices less visible to normal users. Ie, we have one task management system and a problem and change management system baked into one, but normal users (programmers) only use the task management system, while the project manager, tech project manager and CM also have the ability to handle problems and changes in separate workflows.

We’re also adding the ability to have supertickets, where a single problem report can contain several tasks. This is a pseudo development so far, as we’re not actually adding the whole deal right now, we’re just adding the idea of it, not bounds checking or views/reports of it. Basically, every ticket can have a superticket (we add a numeric field to the ticket), which can point to another ticket, which is the “parent” ticket. This makes it possible to handle a large and complex bug in several smaller tickets. Anyways, the idea is there, but it’s not fully implemented. If our management likes it, and the others like it, we could implement it for future usage. I’m worried it’s too complex however. At the same time, one complex system might be better than 6 alltogether different systems as it allows for longer time to learn? Kind of like… well, unix for example. Once you find ls, its a darn good bit faster than having to click your way through a whole heap of paths to find the specified file list.

At the same time, both me and PM are a bit tired of Trac’s shortcomings, maybe change to Mantis for example? My general thought to this however is, we need to stick it out i’m afraid… one more system will just make the normal user less interested in the new tool and hence taking even longer to learn. As it is, people use it at a bare minimum cause they dont know it, give them time to learn it properly, and they might come to like it. Comments on this line of thinking?

For now, tata. Back to writing.

Build components

After a weekend of work, I finally got myself a build component that I’m semi-pleased with, for C and C++ projects, using Subversion. Most likely works for any other lower level programming language as well.

First off, structure. Each component is it’s own BTT(Branches, Tags, Trunk)-root, residing in a Project_Modules directory in subversion. Each component contains an inc, src, test and a stubs directory. Rationale for the BTT-root is that, with a separate BTT-root for each component we can raise the version of each separate component without having to raise it for the entire project.

The Project directory resides on the same level as Project_Modules, and is empty, only containing the subversion property externals pointing to the trunks of the Components in Project_Modules. Rationale for this is to have a simple place to checkout the entire project. It’s a bit dangerous when working with branches, and requires a little bit extra care so one doesnt write into the trunk out of mistake. Possibly block everyone but a specific user to write in the trunks and have that CM person do all the branching/merging. It is time consuming however.

It looks something like this:

• Project_Modules
• Component1
• inc
• src
• test
• stubs
• Component2
• inc
• src
• test
• stubs
• Project

Inc directory is the public interface of the component towards the other components. Src directory contains the actual code of the component. Test contains unit tests (personally, i create a new directory for each new unit test file). Stubs contains the stubs of my own component. Ie, Component1/stubs will contain stubs for the functions in Component1. Rationale being that 95% of the time, we want to stub another component in the same way, instead of keeping stubs of a component in 10 different components, we keep it in one place.

“New” subversion structure using svn:externals

Me and the boss deviced a new structure for the project during the last few weeks, and it’s been slowly refining in our heads until yesterday when we finally implemented it. I think we made a rather refined and complex structure, but once we got it into place physically and once we get the general idea into the developers heads (including me), I think it will prove very powerful.

That being said, I don’t think this is a new structure, I just think people are very quiet about how they use subversion, and it’s a problem. Newcomers do the same old errors over and over again. So, let’s get on to try and explain it all.

Most projects uses a single BTT root, where BTT stands for Branches, Tags and Trunk. Ie, they start a project, and then straight in the root put the BTT, and then inside that, they create the project structure. For example:

• project-root/
• Branches
• Tags
• Trunk
• admin
• src
• out
• test

This is a good basic structure for very small projects, containing perhaps 10′ish files, or where the actual implementation is perfectly homogenous and has no need for separated versioning. Every time we want to make a release, we cheap copy the content to Tags as a new tag (called perhaps /Tags/Milestone1-RC1). We now have a release that we can provide to people.

The problem comes if it isn’t so homogenous. For example, let’s say you are developing a calculator. It has two objects, a numpad and a display. What if you want to make a new version just of the display? You need to make a completely new version, including for the numpad.
Or how about wanting to branch just a small part of the project? Ie, I want to use a branch for the numpad, and then use the trunk for the display. You’d then have to make a cheap copy for the entire tree. Admittedly, it isn’t costing too much.

Our “new” structure deals with this on a different level. Basically, the idea is to have multiple BTT roots, and then use svn:externals to connect the correct tags to create
1) a complete releasable project and
2) a complete workarea project.

For the calculator example, you get the following structure:

• calculator/
• Calculator_Modules/
• Display/
• Branches/
• Tags/
• Trunk/
• Numpad/
• Branches/
• Tags/
• Trunk/
• Calculator/
• Branches/
• Tags/
• Trunk/

As you can see, it looks much more complex, and it is, but the possibilities are infinitely much better.

The Calculator/Trunk/ directory contains a svn:externals property linking in the Calculator_Modules/Display/Trunk as Display and Calculator_Modules/Numpad/Trunk as Numpad. This works by linking external resources into the current directory structure, so basically I would get the trunks into my Calculator trunk, but properly renamed, without them actually being there in the repository. This also works on “real externals” by the way, such as linking in a specific version of a library from some repository on the Internet.

To create a Calculator/Tags/MS1 we could either just set a -rXX to the correct subversion revision, or we would create svn:externals to the correct Display and Numpad Tags, not their trunk. This way, we can say that “Calculator 1.0 contains Display 2.0 and Numpad 2.1″, not “Calculator 1.0 contains Display revision 439 and Numpad revision 587″, or even worse “Calculator 1.0 is revision 587″ which completely lacks granularity.

I’m not completely sure it’s perfect, and others have probably already tested it, but I think it will be pretty sweet .

Stoneage efficiency measurements

Once again I react to the inefficiency of corporate bureaucracy. A very common way of measuring efficiency is to use statistics. How do you use statistics to measure efficiency, and especially in cases such as helpdesk and support functions in a corporation?

A common way of doing this is to just measure closed tickets, and how long time is spent per ticket. I’m not sure how many companies still do this, but I’ve run into it more than once (actually, more times than I can count on my own hands) in the last year.

The problem is that any given person with the sole efficiency measurement being quantitative applied knowingly on him will react by trying to increase his efficiency, and how better to do that than to just cut and paste answers and close any tickets as soon as that’s done.

My response to this, is it quality or quantity that is most interesting? I believe the answer is quite simple in this case, I’m all for quality. Unfortunately, the entire economic system seems to be leaning towards quantity rather than quality these days. I think a half decent compromise however is how CCP in eve-online is handling user support, and it’s the same way I’ve done it when I did user support.

Once you get a problem report (aka ticket), skim through the message, see if you can find a common denominator with some of the most known 6-7 problems (or more if you’re unlucky). Send a cut and paste response (with either an explanation, or a detailed step by step fix), but keep the ticket alive until the person responds. If no response has been received in a given time, say a week, try to contact them again and see if the problem was solved before closing it. If the problem hasn’t been solved by this simple round, it’s either time to send it to 2nd line support, or if there is no 2nd line support start asking for information that can be used to debug the problem, and delve deeper into the problem.

I’m not claiming this is a holy grail of any kind, but it does hopefully keep more customers happy. It requires happy and motivated support personel, and that can at least try to keep face up for a while with customers. To do that, it requires proper motivators, and definitely not some stoneage efficiency measures. Get customer satisfaction index into the game, even if it’s an internal function in your company, or an external function fronting towards your customers.

Dangling locks in the Subversion tree

I recently ran into another rather peculiar and interesting case with subversion. As I’ve already explained, I don’t have actual physical access to the subversion servers we are perusing for our project. We use TortoiseSVN for access to the repository on our local systems.

My boss was working with some excel and word files checked into the repository. He had template.xls locked as he was working in it, he decided to rename it to a template, so template.xlt it is (and checked in). After doing some more work, he decided this was a bad idea, so he changes it back to template.xls, and tries to check it back in again… no luck, it wont work.

15-20 minutes worth of investigation later, we found the problem. TortoiseSVN had been configured to always check “Keep Locks” on commit, hence the client kept the lock when he commited the template.xls deletion. To fix this problem, run svnadmin lslock and then svnadmin rmlock the file in question. Please note that this requires actual access to the physical subversion repository. Http or svnserve will not do in other words.

A step by step on how to cause the Dangling lock problem — All of the actions are done via TortoiseSVN interface:

1. Create a repository.
2. Check out the repository.
3. Add a file to the repository, set the svn:needs-lock property.
4. Commit the file.
5. Lock the file.
6. Change name of the file, and check the “Keep locks” checkbutton.
7. Commit the changes.
8. Change the filename back to the original filename.
9. Commit the change. This step will fail.

This is a rather funny problem imho . We finally managed to get it fixed after contacting the IT department and sending them the command. 10 minutes to cause the problem, 20 minutes to analyze it, 20 hours to wait for the fix. Sometimes there should be less depth in organizations.

Svnsync automatisation via Windows Scheduler

Recently I’ve been setting up for a project, as you might have noticed, and one of the problems we ran into was the fact that we have a centralized subversion repository. We don’t have access to control or install anything on the actual server unfortunately, so hence we can’t install trac on that system, as previously explained. The solution is to sync the main subversion databases to the trac server as previously explained.

A secondary problem arising from this, is that we can’t actually change the hook files of the master repository. The solution to this, is to add a synchronizing scheduler. As I was stressed, and had no time to deal with this properly, I had to run with the Windows Scheduler of all horrible solutions. The following explains how to set it up.

Windows Scheduled synchronize Task

Warning! The windows scheduler does not start automatically when you reboot etc. If you do this, the task restarts at it’s start time, so if it’s set to run at 00.00 every day, and then every 5 minutes, it will restart and start running at 00.00, not as soon as the machine is started up again. This is pure evil.

A scheduled task is set up as follows, create a bat file (ie, c:projectssvnprojectsvnsync-project.bat):

svnsync synchronize http://localhost/svn/project –sync-username slaveuser –sync-password tjohej –source-password password
c:Python25Scriptstrac-admin.exe c:projectstracproject resync

The Second line unfortunately has to be done to resync the new changes with the trac database. After this, create a scheduled task in windows as follows:

1. Go to start -> control panel -> scheduled tasks
2. Click Add scheduled task
3. Click Next
4. Click Browse…
5. Browse to your svnsync-project.bat file, as described above. Double click it.
6. Set name (default is ok). Perform this task: Daily
7. Click Next
8. Start time: 00:00, Every day, Start date: today
9. Fill in username/password to run it as.
10. Select open advanced properties for this task. Click Finish.
11. Go to tab Schedule. Choose Advanced.
12. Click Repeat task, fill in Every 10 minutes, and click Time and fill in 23:59
13. Click OK.
14. Click OK
15. Done.

This could most likely be done from cruisecontrol, but as stated, i’m stressed. I hate windows by now (at/cron is just sooo much nicer to handle).

Next Page »

• Ads by Google

• Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

• Categories

  • Communications
  • Configuration Management
  • Development
  • Economy
  • Editors
  • Frozentux.net
  • General
  • Hardware
  • Ipsysctl
  • Iptables
  • Linux
  • Management
  • Netfilter
  • Personal
  • Phone
  • Projects
  • Robots
  • Ubuntu
  • Uncategorized
  • Video
  • Windows

• Recent Posts

  • Mirabox dist-upgrade causes system to deadlock/hang on bootup
  • Reflashing GlobalScale Mirabox filesystem
  • Ubuntu 12.10 on Dell Precision M4600 APIC issues
  • XBMCBuntu stole my heart
  • Mat Honan hack “cloud isn’t so bad” comment

• Tags

  apt-get C/C++ database Debian Dpkg eclipse embedded FFmpeg GNU Guruplug HTML HTPC iMON JavaScript Joystick KC910 Kino LG Linux lirc Mencoder mirabox mythbuntu mythtv php Player/Stage Player Project Posten presentations pygtk python Qt Remote Control Renoir Robby RP5 server subversion Swedish postal services sync tortoisesvn trac Ubuntu USB VFD zigbee