Bugs, bugs and more bugs
Filed under: Development, Linux, Projects, Ubuntu
Lately, I’ve come to realize more and more that bug handling in open source, and specifically in Ubuntu has dramatically declined in efficiency. For years I’ve been extremely satisfied with using Linux because it’s bug free, there has simply not been any serious bugs that I’ve run into. In the last weeks, I’ve run into several more or less serious bugs in Ubuntu, which got me looking at how the bug handling is done.
First off, a few weeks ago, I ran into a bug with Ubuntu 10.10 Ubiquity (the Live CD installer) where I accidentally marked my old /home drive as ext4 when it was ext3 (but not to reformat it). The installer complied happily, and set it up as ext4, but once it got back online, the harddrive was completely wiped. No warning, no nothing. I started looking around, after a while I’ve found several reports on the same matter on launchpad. For example this and this.
This lead me to take a look at Ubiquity’s other bugs in launchpad, and it’s not very promising. The main installer of Ubuntu 10.10 has 1528 Open bugs as of writing this, of which 846 bugs are new, 35 bugs are marked High importance — and the bugs I found (dare I say, they seem Critical to me, are still not marked with any importance at all). Only 12 bugs are marked as having a patch.
Fine, maybe this is not the poster child of open source. However, the last few days I’ve been severely annoyed by the password popup which is misbehaving. I enter the password, and hit enter (or hit the Authenticate button) and the password field disappears, but the rest of the dialog stays up, and nothing works in it. The only thing you can do is to kill it with the x button. When you do this, you get authenticated…
Since I’m not sure exactly how the authentication is performed in Ubuntu for the update manager etc, I decided to check the update-manager package for Ubuntu on Launchpad. What do I see, if not another package with gigantic mass of bugs filed, but noone dealing with them. 1017 Open bugs, 520 of those are New and 15 marked as High importance. This bug I’ve been having has been reported all over the net, but noone seems to be dealing with it and it isn’t really reported in launchpad. Some computers has it, some doesn’t. It’s nowhere near a critical bug, or even a high importance one, but it’s annoying none the less and it looks extremely crude and comes off giving a fairly unstable feeling.
All this being said, I am wondering how bug handling is done, and how it should be managed on “aggregate” projects such as Debian and Ubuntu. I think the idea is really nice, having upstream bug trackers for each package in the project, but maybe we are spreading too thin having several bug trackers for each minor project? Also, how do we as “normal” users know which package is the reason for the error? I am not so sure it is really the update-manager that is the error in this case, it might as well be some completely other thing behind all that dbus stuff etc. Ie, what is the point of me filing bug reports if I’m not sure they wind up in the right place, or are at all looked after?
Ubuntu 10.10 r8192se_pci driver on the Toshiba Satellite T130-17E (Realtek RTL8191SEvB)
Filed under: Development, Linux, Phone, Ubuntu, Windows
I’ve been home for a few days with a really bad back, and the only thing I’ve been able to do is watch tv, and some minor work with the laptop. I’ve been running Windows 7 which it was delivered with for a few months to get a feel of it, and I must say I was pleasantly surprised for the most part. It hasn’t crashed more than once or twice in three months, and it is fairly snappy (except boottimes seems to get worse and worse, at this point it takes 2-3 minutes to boot). Anyways, for some reason I get a bit sad inside (and bored) every time I boot Windows, there is just “something” about the feel, the look or… I can’t really put my finger on it, that I can’t stand. How the windows open and close perhaps, I just don’t know.
So, yesterday I wanted to test android SDK, re-realized just how much of a bitch it is to install stuff on Windows, so I finally got around to installing Ubuntu 10.10 on it (Already running 10.10 on desktop and mediapc), removing the extra backup partition they deliver laptops with these days. Sidenote, isn’t it a bit like selling a candle with a flammable fire extinguisher to sell a laptop with 500gig harddrive, split it in half, and use on half for “backups”? I digress.
So, installation went almost flawless. The wireless card was identified, saw networks, but was unable to connect to any of them. I managed to pass the installation using trusty old cables, and after installation was done I started fiddling about, reading on the net etc, and found noone who had solved the combination or at least written about it.
Main problem seems to have been hardware wep encoding/decoding, which can be turned off using the hwwep flag to the r8192se_pci module. On Ubuntu 10.10, remove the module, and then reload it by doing this:
rmmod r8192se_pci
modprobe r8192se_pci hwwep=0
If your network works now, you can automate the setting by editing/adding the configuration to modprobe.d, by editing /etc/modprobe.d/realtek.conf and adding the following line:
options r8192se_pci hwwep=0
I hope this has been some help!
Guruplug arrived, sounds like a jet
The first few paragraphs here are rather harsh vs the Guruplug that I received, and yes, GlobalScale deserves some really bad critique for how this has been handled, but please read the final parts to get a full picture. In all honesty, sometimes I think I should just change name to “Grumpy Fart”.
I finally received the Guruplug Server Plus last week that I ordered 3 months ago. My first reaction is a big WTF on this machine. Apparently, they’ve had big troubles with overheating in the Guruplug, so badly so that a lot of units died from it. Someone over at GlobalScale Technologies had the absolutely idiotic idea to put a fan into the machine. It’s not just any fan, it’s a 2 cm maglev fan running at 3-4000 rpm, and no powermanagement whatsoever and it is directly hooked up to the power source inside the Guruplug, hence it will not be possible to vary the rpm, ever, without a hardware hack. Also, the fan is absolutely horribly placed as is evident from several pictures on the net (and by opening the machine, voiding the warranty) — it is placed with 80-90% of the back of the fan covered by a metal plate (the two gigabit ethernet interfaces), and when the machine is closed up as delivered, it has a big plastic plate (power source cover) covering the front of the fan, with less than 2 mm clearance. All this means is that it is incredibly noisy (easily 30-40dB, way louder than my core 2 quad machine with 8 gig ram and 4 fans in it, while in bootup and before the nvidia graphics card has gone into power management), and has close to no effect at all.
I bought this machine when there was no fan, and there wasn’t even talk of a fan or any mails to acknowledge this design change, so I was heavily inclined at sending the thing back for a refund, but I realized that with my normal luck, it would take several weeks to find a new machine matching my needs, and then another few months before receiving it. Because of this, I winded up simply ripping out the fan, and voiding the warranty. So far so good, and nothing bad has happened. The plug is still running approximately at the same temperature as before, but slightly higher, and I hope there will be no ill effects.
Second WTF when I finally got the machine started up was the installation — first off it is a really nice debian install, I was prepared to start straight off with the jtag interface and installing images and reinstalling etc etc, but there was a perfectly working machine there, straight off. Nice. I thought. Then came the wtf moment, the machine booted up and set an ip address for the uap0 interface (wifi access point interface) to 192.168.1.1. My plan is to use it hooked up to a wired network via gigabit, so I went straight at it, added the eth0 interface to /etc/network/interfaces with a static ip, delete routes and ip’s for uap0 and it worked… for an hour or so, then it stopped working. Reboot, still not working, notice that uap0 is back at it’s old location 192.168.1.1 and routes are back, but nothing in the bootup scripts. Finally find out they have a /etc/init.d/rc.local file point to a /root/init_setup.sh, which in turn does a heap of stuff — including setting up network parameters etc, overriding the normal configuration parameters in a nonstandard way.
Personally, I find this type of “hacked” environments to be despicable. For a private system in your home, fine do whatever you wish, but when it is a public server at a company, or something that you sell to customers, you wind up with a product that the other admins and/or customers can not trust the setup of. Luckily, the whole system is at least included in installed .deb packages. The init_setup.sh, together with the fact that there is no way of recovering if you do a simple screw up of the network setup without the JTAG interface is a major drawback imho.
So… that is the “bad” part of my experience so far. At this point I was rather underwhelmed, but I have been pleasantly surprised by the performance of the little bugger, and except for the startup scripts, it is rather nicely installed from what I have see so far. It would be interesting to gather up a complete list of the “hacks” they have performed to get it all together. The machine is currently able to do exactly what I set out to do, in less than a 2 days of configuration/fixing/fiddling about/etc. I didn’t manage to get the “auto connect” of my usb disk to work, but mount commands work and I didn’t really look at how it’s supposed to work (might be that you need to reboot machine etc). The idea with the machine on my part, is to set it up as a 24/7 machine, replacing my other machine doing this work, but at a lower power consumption. So far, the machine is running the following services for me:
- DHCP
- Dynamic DNS
- Samba and NFS filesharing
- Ssh network login server
- USB Disk/”NAS” function
- Bittorrent (transmission-daemon with transmission-remote on all other machines)
Additionally I hope to use it for the following as well:
- Printer server
- Zeroconf/avahi
- Firewall (possible use as a portable firewall?)
- Temperature sensors etc via GPIO?
- Bluetooth, still haven’t figured out what to do with this…
As you can see, the machine is very capable, and all at a very low power consumption of <5W compared to my old computer doing all this, running at 170W approximately. At current power costs, I am expecting to have made up the money I spent on the Guruplug within less than 9 months. Let’s just keep fingers crossed that the machine wont die of heat before that 🙂 .
Conclusion
In conclusion, I am really afraid to say anything but this, I can not really recommend this product, not unless you’re either ready to do some serious hacking, or you plan to run it in a garage or some such place. A wardrobe or closet is simply not enough, it sounds too much as it is. The idea and the thought behind the machine is great, I just wish the execution of it was as good.
Shaving some time
I was talking to a colleague the other day and got into a discussion regarding some ways of coding c++, such as whether to use division or bit shifting — almost all modern compilers can optimize this so it’s basically just a matter of what looks best. At the same time, we got into a minor discussion regarding references and pointers in c++. I made a small test and found some rather amusing results, which is quite obvious once you think about it, but still very scary considering how common it is to use the pointer construct:
// Compile using g++ -lrt -o lala lala.cpp
#include <iostream>
#include <time.h>
int* lala_ptr()
{
int *y = new int;
*y = 5;
return y;
}
int& lala_ref()
{
int x =5;
int &y = x;
return y;
}
timespec diff(timespec start, timespec end)
{
timespec temp;
if ((end.tv_nsec-start.tv_nsec)<0) {
temp.tv_sec = end.tv_sec-start.tv_sec-1;
temp.tv_nsec = 1000000000+end.tv_nsec-start.tv_nsec;
} else {
temp.tv_sec = end.tv_sec-start.tv_sec;
temp.tv_nsec = end.tv_nsec-start.tv_nsec;
}
return temp;
}
int main(int argc, char **argv)
{
timespec time1, time2, time3;
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time1);
for (unsigned int i=0;i<(unsigned int)-1;i++)
{
int &z = lala_ref();
}
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time2);
for (unsigned int i=0;i<(unsigned int)-1;i++)
{
int *z = lala_ptr();
delete z;
}
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time3);
std::cout << "Reference diff(" << time1.tv_sec << ":" <<
time1.tv_nsec << ", " <<
time2.tv_sec << ":" <<
time2.tv_nsec << ") = " <<
diff(time1, time2).tv_sec << ":" <<
diff(time1, time2).tv_nsec << std::endl;
std::cout << "Pointer diff(" << time2.tv_sec << ":" <<
time2.tv_nsec << ", " <<
time3.tv_sec << ":" <<
time3.tv_nsec << ") = " <<
diff(time2, time3).tv_sec << ":" <<
diff(time2, time3).tv_nsec << std::endl;
}
Below is a sample of the output generated by the test code above:
oan@laptop4:~/Projects/test$ ./testRefVsPointer Reference diff(0:3869272, 25:234466470) = 25:230597198 Pointer diff(25:234466470, 299:547382527) = 274:312916057
So, the question for you all, can you figure out what's wrong? 😉
Screenmovie 1.1 released
Filed under: Development, Frozentux.net, Linux, Ubuntu, Video
A quick note that Screenmovie 1.1 has been released. It’s still very crude, but adds some sound recording and the ability to turn it on/off. Postprocessing is not supported yet but should be there in the next version.
Features:
- Record video
- Record sound
- Configure file format
- Configure video codec + settings (5-6 codecs chosen for now)
- Configure audio codec + settings (2 codecs for now)
I still have some problems, but I just found some info on how to possibly make some problems better at least.
Todo:
- Fix some high cpu usage problems
- Add global keybindings
- Postprocessing encoding
- Clean up and add some values specific for codecs (as required).
Unit testing and stubbing singletons
I got a bit curious about stubbing Singletons for testing during the weekend as well. We often find ourselves needing to test large codebases at work, and in the current project I’m in, we do complete end to end signal flow tests, but people are finally realizing that this will simply not do. For this reason, we’re doing a lot of work to try to split the entire project up into manageable chunks. One of the main problems has been the incessant use of singletons. A simply half-way-there to doing full out interfaces is to simply make all public function calls virtual and then create a stub class of the singleton saving the message or whatever passed on, into a variable which can be grabbed and tested from the actual unit test.
A sample of the general idea below:
#includeclass A { public: static A *instance() { std::cout << "A::instance()" << std::endl; if (!s_instance) s_instance = new A; return s_instance; }; A() { std::cout << "A::A()" << std::endl; }; // Virtual makes the difference virtual void send(int i) { std::cout << "A::send()" << std::endl; // Blah blah, send i or something }; static A *s_instance; private: }; class stub_A: public A { public: static stub_A *instance() { std::cout << "stub_A::instance()" << std::endl; if (!s_instance) { s_instance = new stub_A; A::s_instance = s_instance; } return s_instance; }; stub_A() { std::cout << "stub_A::stub_A()" << std::endl; }; void send(int i) { std::cout << "stub_A::send()" << std::endl; y = i; }; int getMessage() { return y; }; private: int y; static stub_A *s_instance; }; A *A::s_instance = 0; stub_A *stub_A::s_instance = 0; int main(int argc, char **argv) { stub_A::instance()->send(5); std::cout << "stub_A::instance()->getMessage() == " << stub_A::instance()->getMessage() << std::endl; A::instance()->send(7); std::cout << "stub_A::instance()->getMessage() == " << stub_A::instance()->getMessage() << std::endl; }
Python multiprocessing
Sometimes I find a good time by writing small but easy to understand test programs to research specific behaviours in some language or another. Sometimes, the programs grows a bit wieldy and not so easy to understand, but all’s good that ends well. During the last year or so, I’ve grown more and more interested in python programming and finding it very enjoyable. There are some strange constructs that can be hard to wrap your head around, and sometimes I run into some very weird problems, but it’s not a big problem (imho).
My last forays has been into multiprocessing and how it behaves in Python. One of the features I had a hard time to wrap my head around since there are some syntactical weirdness that could use some addressing, or at least takes a bit of time to get your head around.
- Multiprocessing requires all objects to be pickled and sent over to the running process by a pipe. This requires all objects to be picklable, including the instance methods etc.
- Default implementation of pickling functions in python can’t handle instance methods, and hence some modifications needs to be done.
- Correct parameters must be passed to callbacks and functions, via the apply_async. Failing to do so causes very strange errors to be reported.
- Correct behaviour might be hard to predict since values are calculated at different times. This is especially true if your code has side effects.
The small but rather interesting testcode below explores and shows some of the interesting aspects mentioned above. Of special interest imho is the timing differences, it clearly shows what you get yourself into when doing multiprocessing.
#!/usr/bin/python
import multiprocessing
def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
return _unpickle_method, (func_name, obj, cls)
def _unpickle_method(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(obj, cls)
import copy_reg
import types
copy_reg.pickle(types.MethodType,
_pickle_method,
_unpickle_method)
class A:
def __init__(self):
print "A::__init__()"
self.weird = "weird"
class B(object):
def doAsync(self, lala):
print "B::doAsync()"
return lala**lala
def callBack(self, result):
print "B::callBack()"
self.a.weird="wherio"
print result
def __init__(self, myA):
print "B::__init__()"
self.a = myA
def callback(result):
print "callback result: " + str(result)
def func(x):
print "func"
return x**x
if __name__ == '__main__':
pool = multiprocessing.Pool(2)
a = A()
b = B(a)
print a.weird
print "Starting"
result1 = pool.apply_async(func, [4], callback=callback)
result2 = pool.apply_async(b.doAsync,
[8],
callback=b.callBack)
print a.weird
print "result1: " + str(result1.get())
print "result2: " + str(result2.get())
print a.weird
print "End"
The above code resulted in the following two runs, and if you look closely, the timing problems show up rather clearly. Things simply don’t happen in the order always expected when threading applications:
oan@laptop4:~$ ./multiprocessingtest.py A::__init__() B::__init__() weird Starting weird func B::doAsync() callback result: 256 B::callBack() 16777216result1: 256 result2: 16777216 wherio End oan@laptop4:~$ ./multiprocessingtest.py A::__init__() B::__init__() weird Starting weird func B::doAsync() callback result: 256 B::callBack() 16777216 result1: 256 result2: 16777216 wherio End
One more warning is in order. A job that leaves via the multiprocessing.Pool, and then calls the callback function has a major effect that could take some getting used to. The callback is run in such a fashion that if a class was changed, the change has not taken place in the context of the callback.
Regarding web browsers and User Interfaces
Filed under: Development, Frozentux.net, Linux, Windows
I had a discussion this morning with my wife regarding her new computer (Toshiba laptop). A few days back, I was contemplating the possibility to try and get rid of the Windows 7 license it came with the EULA disagreement argument and get my money back, but after much consideration and realizing that her school (Chalmers University) pretty much requires her to run Windows 7 and Office 2007 (all applications used are windows based with Unix/Linux alternatives only irregularly, and her lab professors more or less frowns on OpenOffice.org vs MS Office formatting problems), we decided to keep it. This is another story, how Chalmers went from being a very open University to a complete clusterfuck of a Microsoft bootcamp and something I’d love to write more about at some point, but not now.
The computer has arrived finally, and my wife had a little time to play around with Windows 7. I’ve personally not used Windows since Win XP and refuse to lay hands on it unless “forced” to at work. Her main gripes came with Internet Explorer 8, and I couldn’t help but laugh very loudly at her assessment of it. It incessantly keeps asking questions and giving hints and ideas like “How do you like this?”, “Did you know that IE plugin blah …?”, “We can help you do …?”, “Do you have herpes?” to the point where the only sane comparison she could make is a very horny and needy drunk boy at a club. It took her less than one battery charge to switch back over to Firefox. My guess is that they simply made the user interface a bit … too helpful, and tried to be too reactive to “possible needs in this context”.
So, on this topic, I thought it could be slightly interesting to see this months statistics and split between different operating systems and web browsers.
Operating systems
I can’t really say that this site is representative of the world in general, but I think it is interesting to see the level of windows users. I’m guessing this is a sign that many of the people visiting are reading about iptables and using linux for servers and firewalls, but aren’t using it as a desktop system (yet?).
| Windows | 59.1 % |
| Linux | 31.3 % |
| Macintosh | 6.4 % |
| Unknown | 2.7 % |
Web browser
Personally, I find this part the most astounding. That IE has gone so low is simply not something I had expected, and how big market share firefox has… incredible.
| Firefox | 60.7 % |
| MS Internet Explorer | 16.4 % |
| Safari | 11.4 % |
| Mozilla | 4.5 % |
| Opera | 4.1 % |
| Unknown | 1.2 % |
I will try to do some more work on this in the future, could be interesting to see this change over time. I know the reliability of my own source isn’t excellent and the error margin is probablythrough the roof, but it should be rather fun to dig in to.
Screenmovie released
My first application ever in python has been released, Screenmovie as I chose to call it. It’s a very simple application for recording your desktop or windows on your desktop. Download and put somewhere in your filesystem and unpack it, and start it. It’s very simple to use, it uses pygtk to create a taskbar icon in your window manager. Click it once, and you get a cross-hair mouse pointer. Click the window you want to record, and the application starts recording straight off. Click the icon again to stop recording. To record the entire desktop, click the background somewhere.This application only works with Linux and other systems using X11 window management and requires xwininfo, ffmpeg and python 2.6 + pygtk+.
If you have any comments or thoughts, feel free to mail me.
Secondly, I’ve come quite far along with the pyEveApi, a rather large project I began a few weeks back. The project is perfect imho, it gives enough problems to solve and also introduces me to a lot of functionality in Python that I would have a hard time finding out about otherwise. I’m not close to a release yet as I only support 4 API’s so far, and just restarted quite a lot of the work with rewriting the caching functionality. The Long caching method has been fairly well implemented atm, and works, and I’m currently working on the Modified Long method, using the WalletTransactions API to test with. Either way, I’m hoping to have something releasable sometime soon.
Working with Python
A while back I started pondering Python as a language, and mostly the fact that I haven’t really tried any of the newer and more modern languages around. Not to any great lengths at least. So, about a year or two ago I started fiddling with it, mostly just trying different scripts, writing some small hacks, and so forth.First, I wrote a Bluetooth scanner for bluez, then via Dbus interface and so forth. The ease this came with was completely amazing tbh.
Last few weeks I started writing some larger applications with python, to try and learn it a bit better. First of all, I’ve had large problems with all desktop recording tools I’ve found so far — they either don’t support GL/Direct rendering, or they simply crash, or colors are all flunked out, and so forth. So, simple idea was to use the tool I know that works — ffmpeg and xwininfo — and make an easy to use interface for it, a taskbar icon. Click it once, and get a cursor to choose which window to record, click it and recording starts (icon changes to a recording button), and click it again to stop it. There is a right click menu to make some preferences and to stop the recorder as well. It works, and it’s very very simple :-).
Second set of tools I decided to make is a set of API’s for the Eve Api. This is also nothing new, but it was a good way to get a handle on how Python works, and how the Eve Api works. On top of this I have written some very simple apps that downloads all marketorders and wallettransactions in game and displays them nicely in a console. It has built in support of all caching styles, downloading, login, local disk cache, and so forth, but so far has very limited support for the different API’s.
My main observation so far has been that I like Python for it’s hackability and ease of use. Stuff comes naturally once you get used to it, but the dynamic typing and hackability also comes at some price. First of all, it is very easy to create scenarios where the code will crash due to bad types in the dynamic type checking, and the hackability also means you need to be very wary of your architecture. Hacking away is almost never a good thing if you are preparing to undertake any larger projects imho, the risk of writing bad code is paramount, and testing bad code … well, it’s just worse 😉 . That said, I do like the language, it’s completely awesome, and someone else said not so long ago, you can get around the problems, you just have to test the code so much harder for it to be reliable. Personally, I wouldn’t use Python for a critical system/application, but for my day to day use, it’s not a problem, I can survive a crash every once in a while. Also, after getting used to the pydev plugin to eclipse, I just love the entire environment even more :-).