Python multiprocessing
Sometimes I find a good time by writing small but easy to understand test programs to research specific behaviours in some language or another. Sometimes, the programs grows a bit wieldy and not so easy to understand, but all’s good that ends well. During the last year or so, I’ve grown more and more interested in python programming and finding it very enjoyable. There are some strange constructs that can be hard to wrap your head around, and sometimes I run into some very weird problems, but it’s not a big problem (imho).
My last forays has been into multiprocessing and how it behaves in Python. One of the features I had a hard time to wrap my head around since there are some syntactical weirdness that could use some addressing, or at least takes a bit of time to get your head around.
- Multiprocessing requires all objects to be pickled and sent over to the running process by a pipe. This requires all objects to be picklable, including the instance methods etc.
- Default implementation of pickling functions in python can’t handle instance methods, and hence some modifications needs to be done.
- Correct parameters must be passed to callbacks and functions, via the apply_async. Failing to do so causes very strange errors to be reported.
- Correct behaviour might be hard to predict since values are calculated at different times. This is especially true if your code has side effects.
The small but rather interesting testcode below explores and shows some of the interesting aspects mentioned above. Of special interest imho is the timing differences, it clearly shows what you get yourself into when doing multiprocessing.
#!/usr/bin/python
import multiprocessing
def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
return _unpickle_method, (func_name, obj, cls)
def _unpickle_method(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(obj, cls)
import copy_reg
import types
copy_reg.pickle(types.MethodType,
_pickle_method,
_unpickle_method)
class A:
def __init__(self):
print "A::__init__()"
self.weird = "weird"
class B(object):
def doAsync(self, lala):
print "B::doAsync()"
return lala**lala
def callBack(self, result):
print "B::callBack()"
self.a.weird="wherio"
print result
def __init__(self, myA):
print "B::__init__()"
self.a = myA
def callback(result):
print "callback result: " + str(result)
def func(x):
print "func"
return x**x
if __name__ == '__main__':
pool = multiprocessing.Pool(2)
a = A()
b = B(a)
print a.weird
print "Starting"
result1 = pool.apply_async(func, [4], callback=callback)
result2 = pool.apply_async(b.doAsync,
[8],
callback=b.callBack)
print a.weird
print "result1: " + str(result1.get())
print "result2: " + str(result2.get())
print a.weird
print "End"
The above code resulted in the following two runs, and if you look closely, the timing problems show up rather clearly. Things simply don’t happen in the order always expected when threading applications:
oan@laptop4:~$ ./multiprocessingtest.py A::__init__() B::__init__() weird Starting weird func B::doAsync() callback result: 256 B::callBack() 16777216result1: 256 result2: 16777216 wherio End oan@laptop4:~$ ./multiprocessingtest.py A::__init__() B::__init__() weird Starting weird func B::doAsync() callback result: 256 B::callBack() 16777216 result1: 256 result2: 16777216 wherio End
One more warning is in order. A job that leaves via the multiprocessing.Pool, and then calls the callback function has a major effect that could take some getting used to. The callback is run in such a fashion that if a class was changed, the change has not taken place in the context of the callback.
Regarding web browsers and User Interfaces
Filed under: Development, Frozentux.net, Linux, Windows
I had a discussion this morning with my wife regarding her new computer (Toshiba laptop). A few days back, I was contemplating the possibility to try and get rid of the Windows 7 license it came with the EULA disagreement argument and get my money back, but after much consideration and realizing that her school (Chalmers University) pretty much requires her to run Windows 7 and Office 2007 (all applications used are windows based with Unix/Linux alternatives only irregularly, and her lab professors more or less frowns on OpenOffice.org vs MS Office formatting problems), we decided to keep it. This is another story, how Chalmers went from being a very open University to a complete clusterfuck of a Microsoft bootcamp and something I’d love to write more about at some point, but not now.
The computer has arrived finally, and my wife had a little time to play around with Windows 7. I’ve personally not used Windows since Win XP and refuse to lay hands on it unless “forced” to at work. Her main gripes came with Internet Explorer 8, and I couldn’t help but laugh very loudly at her assessment of it. It incessantly keeps asking questions and giving hints and ideas like “How do you like this?”, “Did you know that IE plugin blah …?”, “We can help you do …?”, “Do you have herpes?” to the point where the only sane comparison she could make is a very horny and needy drunk boy at a club. It took her less than one battery charge to switch back over to Firefox. My guess is that they simply made the user interface a bit … too helpful, and tried to be too reactive to “possible needs in this context”.
So, on this topic, I thought it could be slightly interesting to see this months statistics and split between different operating systems and web browsers.
Operating systems
I can’t really say that this site is representative of the world in general, but I think it is interesting to see the level of windows users. I’m guessing this is a sign that many of the people visiting are reading about iptables and using linux for servers and firewalls, but aren’t using it as a desktop system (yet?).
| Windows | 59.1 % |
| Linux | 31.3 % |
| Macintosh | 6.4 % |
| Unknown | 2.7 % |
Web browser
Personally, I find this part the most astounding. That IE has gone so low is simply not something I had expected, and how big market share firefox has… incredible.
| Firefox | 60.7 % |
| MS Internet Explorer | 16.4 % |
| Safari | 11.4 % |
| Mozilla | 4.5 % |
| Opera | 4.1 % |
| Unknown | 1.2 % |
I will try to do some more work on this in the future, could be interesting to see this change over time. I know the reliability of my own source isn’t excellent and the error margin is probablythrough the roof, but it should be rather fun to dig in to.
Comments requiring login
Due to the amount of spam I receive in the comments fields of this site, I decided to turn on the requirement for all users to be logged in to post any comments. So far, there has been 770 spams posted and only 38 real comments. It’s easy to clean the spams out for sure (akismet, normal users don’t even have to see them), but just seeing the names of some of the links spammed are making me sick.
As a sidenote, with todays climate, maybe I shouldn’t say this, but if I ever ran into a person with the “quirkiness” required to watch (or for that matter produce) what is spammed these days, I’d probably go completely ballistic and on a killing spree. I definitely don’t condone the type of filtering done and/or tested on national levels, such as in Australia and Germany, but I can definitely understand why less hardened people turn to those extremes. My main reason for disliking that filtering isn’t really the content it is supposed to target, but rather where it is heading. The step from no filtering to filtering extreme sex is much larger than the step from extreme sex to “rough” sex, and who will draw the line in the end?
What is the difference between extreme and normal? Most humans like to see things in black and white, and we forget about the greyscales. This is trait easily manipulated by people with an agenda (christian and muslim extremists, politicians, law enforcements and so forth) by pretty much saying “Look here, look at that absolute abomination of a thing going on, we can not allow this to happen to XXX. Let’s do something, let’s ban everything remotely connected to it.” In all honesty, that shit scares the hell out of me. Not so much the rethoric used, but the fact that so many falls for it. This whole 0-tolerance, black and white and fight or flight mentallity.
Anyways, just some random thoughts on the matter.
Screenmovie released
My first application ever in python has been released, Screenmovie as I chose to call it. It’s a very simple application for recording your desktop or windows on your desktop. Download and put somewhere in your filesystem and unpack it, and start it. It’s very simple to use, it uses pygtk to create a taskbar icon in your window manager. Click it once, and you get a cross-hair mouse pointer. Click the window you want to record, and the application starts recording straight off. Click the icon again to stop recording. To record the entire desktop, click the background somewhere.This application only works with Linux and other systems using X11 window management and requires xwininfo, ffmpeg and python 2.6 + pygtk+.
If you have any comments or thoughts, feel free to mail me.
Secondly, I’ve come quite far along with the pyEveApi, a rather large project I began a few weeks back. The project is perfect imho, it gives enough problems to solve and also introduces me to a lot of functionality in Python that I would have a hard time finding out about otherwise. I’m not close to a release yet as I only support 4 API’s so far, and just restarted quite a lot of the work with rewriting the caching functionality. The Long caching method has been fairly well implemented atm, and works, and I’m currently working on the Modified Long method, using the WalletTransactions API to test with. Either way, I’m hoping to have something releasable sometime soon.
Working with Python
A while back I started pondering Python as a language, and mostly the fact that I haven’t really tried any of the newer and more modern languages around. Not to any great lengths at least. So, about a year or two ago I started fiddling with it, mostly just trying different scripts, writing some small hacks, and so forth.First, I wrote a Bluetooth scanner for bluez, then via Dbus interface and so forth. The ease this came with was completely amazing tbh.
Last few weeks I started writing some larger applications with python, to try and learn it a bit better. First of all, I’ve had large problems with all desktop recording tools I’ve found so far — they either don’t support GL/Direct rendering, or they simply crash, or colors are all flunked out, and so forth. So, simple idea was to use the tool I know that works — ffmpeg and xwininfo — and make an easy to use interface for it, a taskbar icon. Click it once, and get a cursor to choose which window to record, click it and recording starts (icon changes to a recording button), and click it again to stop it. There is a right click menu to make some preferences and to stop the recorder as well. It works, and it’s very very simple :-).
Second set of tools I decided to make is a set of API’s for the Eve Api. This is also nothing new, but it was a good way to get a handle on how Python works, and how the Eve Api works. On top of this I have written some very simple apps that downloads all marketorders and wallettransactions in game and displays them nicely in a console. It has built in support of all caching styles, downloading, login, local disk cache, and so forth, but so far has very limited support for the different API’s.
My main observation so far has been that I like Python for it’s hackability and ease of use. Stuff comes naturally once you get used to it, but the dynamic typing and hackability also comes at some price. First of all, it is very easy to create scenarios where the code will crash due to bad types in the dynamic type checking, and the hackability also means you need to be very wary of your architecture. Hacking away is almost never a good thing if you are preparing to undertake any larger projects imho, the risk of writing bad code is paramount, and testing bad code … well, it’s just worse 😉 . That said, I do like the language, it’s completely awesome, and someone else said not so long ago, you can get around the problems, you just have to test the code so much harder for it to be reliable. Personally, I wouldn’t use Python for a critical system/application, but for my day to day use, it’s not a problem, I can survive a crash every once in a while. Also, after getting used to the pydev plugin to eclipse, I just love the entire environment even more :-).
Final thoughts on the embedded Linux seminar
The embedded Linux seminar was held last week, and in general I feel that it went pretty well in line with my expectations. It’s been a long time since I held any real presentations, so the first two presentations in Gothenburg I was very much nervous and lost myself a few times along the line. The two presentations in Stockholm and Oslo went surprisingly well however, and I don’t think I made any huge errors from a pure presentation point of view.
Licensing
Now, that said, we did run into a few snags and after pondering what we can learn from the whole presentation/seminar — and there are a few points I’d like to raise both for the attendees and anyone else who might be interested. Being an “engineer”, I like to consider what went wrong, etc. Most points are of minor interest, but one of the absolute major points that we really didn’t get across properly, licensing issues with open source, or rather, licensing issues are manageable, however this was not the main area that we (or at least me) where there to talk about. Our main error was simply that we forgot communicating properly with each other, and correlate what we where saying. Also, this seminar wasn’t really about licensing issues, Nohau has an entire seminar/course on that topic alone, and you could easily fill out an entire university course on open source licensing.
The main point I tried to get across was that, yes, you need to be wary about licenses and you need to look at what is required of you, but that’s nothing different from any closed source licenses either, and you should be putting policies as well as processes in place to handle it, and push knowledge on how to handle licenses must be disseminated throughout the project.
So, to address some of the main licensing questions we received:
- No you will not have to give away your code if you link the code properly.
- You will have to set up proper procedures to handle any third party sources.
- You will have to create processes for everyone to follow to get any third party sources “accepted”.
- You will have to adhere to third party licenses, if you don’t, be prepared to be forced to and receive some bad publicity for it. (A lot of companies/people do get away with it, but is it worth risking it?)
Reliability
The second large question we got was, when would you use Linux, and when wouldn’t you use it, in a life or death situation? Simply put, I wouldn’t put it in a system where a person or persons would die if the process/hardware/appliance crashes, but that’s me. I would generally speaking make the life supporting/critical system run on a separate hardware, and then make all the critical stuff run in that context/hardware, and then a second piece which communicates with the critical hardware and do the higher end “stuff” that might be interesting (communicate to centrals, user interfaces, settings, etc). This way, the critical stuff can be kept simplistic and reliable (in my experience, reliability is a function of complexity, the more complex, the higher the failure rate).
In most projects, this has to be decided on a case by case basis, and due diligence must most of the time be taken with the laws and standards of each area. What is possible and advisable to do in house and home automation is not the same as in airplanes or medical systems for pretty obvious reasons.
Presentation depth/breadth/focus
Finally, a minor point, I got some criticism for being too shallow, not going enough in depth. Well, I could have stayed on discussing tool-chains and how to make one for hours, or I could have talked entirely about Linux internal boot order and why it works the way it does, but that wasn’t the goal of the entire seminar. That stuff could be studied to death, in the end your better off “getting” the top-down structure of an embedded Linux project and then just experiment on your own rather than get everything served in forum that can not make justice to everyones requirements. Next time however, I will try to maintain a deeper focus on a bit fewer topics, or get more time to speak in.
Anyways, I think it was fun and a huge experience. I hope most people visiting found the seminar interesting and had something out of it.
Website downtime
Filed under: Frozentux.net, General, Personal
As some people may have noticed, the servers was down for almost a week. This was due to several hardware problems at my host, which where very unfortunate. When I put up the new webpage a few weeks ago, we also moved the daemons/hosting to a new one running on a new machine. This machine has been misbehaving for several weeks, and we switched over to another identical machine to rule out hardware error. In the end, we managed to get it solved. Last week both servers where killed due to power fluctuations (I believe if I interpreted it correctly). As a replacement, two new machines where ordered in, and they where both DOA (Dead on arrival). On top of this, I was told they received a faulty 20x500gb harddrive batch a few months back, which has had a 50% failure rate so far.
All this being said, I want to also extend my gratitude to these people for hosting this site, and for the wonderful work they do in running it. In 7 years of hosting this domain/host, I have rarely had a single breakdown, and this is the first time any major problem has existed with the hosting. In short, the wonderful fellows in the Netherlands that keeps hosting this site has had a really rough week, and I really wish them the best of luck getting everything back in order. Until then, please be patient with the site if it’s down or something is not working correctly, or even better, throw a mail in my general direction.
Using Ubuntu as Media Server for Xbox360
I got a Xbox360 since about a year and I just noticed it had some way of connecting to a PC, using the PC as a Media Server. Unfortunately it required a Windows Media Center installation to work, or so it claimed at least. This is probably not news to anyone, but it was very easy to get Ubuntu (or any other Linux distro as a matter of fact) to serve media for the Xbox 360. Xbox 360 uses UPnP to get media from the Windows Media Center PC. To make any recent Ubuntu able to serve UPnP suitable for the Xbox, do the following:
- sudo apt-get install ushare
- sudo dpkg-reconfigure ushare
- sudo vim /etc/ushare.conf
- Make sure all the settings are correct.
- sudo vim /etc/default/ushare
- Make sure it contains USHARE_OPTIONS=”–xbox”.
- sudo /etc/init.d/ushare restart
You should now be able to find the PC by searching for it from the Xbox interface (the name you set in ushare.conf should show up in the list of found PC’s). Now that that’s said, I should hint that the Xbox360 has a really shitty availability of audio and video codecs, and I don’t know if it’s possible to resolve this problem. There are hints that there is something called a UPnP Media Adaptor on the ushare website which should be able to convert to proper file formats as necessary, but ushare does not have that ability. Of course, that would give a shitload of cpu load on the fileserver as well, something which sounds less good in my opinion.
My personal opinion so far, Xbox 360 media center is really simple to use, but the available codecs, flexibility and scalability is catastrophically bad in comparison to my MythBuntu installation (still running 9.04 though). The Mythbuntu installation is a bit heavy on the configuration, but much more flexible, handles almost all codecs I’ve run into without even a hitch, and very scalable.
Thoughts on Intellectual Property
A few days ago I watched a show on swedish television where two doctors in IP and creativity went visiting badly running companies and helping them get creativity back in to the company, and hence making the badly running companies back in game. The methods they used seemed fairly straight forward and I can’t really say much about them, except it seemed to be how it should be done. Bring in the people who matters, talk things over, get ideas for new business areas, get ideas on how to improve efficiency in the employees work, how to run the company, new retailable products and services, and so forth. All this is just perfect, however, there was one comment on a subject that struck me hard.
They where complaining that Sweden is one of the countries putting the most money into research and development, yet we get the least IP out of it, and for some reason they could not understand why. I work with development, and to some extent research and I’m not the least bit shocked about this. Lack of cooperation, simple as that. Humans are in a sense made to cooperate, to work as a team and to throw ideas back and forth, especially in this day when the mass of information is simply impossible to keep tabs on. There are probably more research and development papers published every day than I could read in an entire lifetime and I will never be able to read more than a small bit of everything out there in my specific area of expertise, and not even there will I be able to read everything. The problem stems, in my humble opinion, from all the IP lawyers and lobbyists getting their messages across all too well.
Let me give an example of what I mean, for the last 3-4 years, I’ve worked on 4 different projects at 4 companies. At every single one of them, we have all been brought to a big meeting with the entire project/department/company where some “IP department” representative or lawyer has stood up and told us, whenever you get an idea:
- Shut up about it, don’t talk to anyone.
- Send an e-mail (or preferably sealed envelope or actually come talk to them in person) about the idea to the IP people.
- They’ll discuss the idea with you, then wait for half a year or so.
- If the idea is patentable, wait another 2-3 years for the patent to go through.
- Now you can implement and start talking about the patent, and hopefully cash in on all the suckers who re-created your patent while you where biding your time.
I hope you can see the problem here. The whole idea is so completely remote from how creativity for most of us normal people work that it’s not even funny. For my own part, the best and most inventive days I’ve had, has been with friends. During my education, for example, we went out by the sea and talked mostly crap, but this is where and when we cross-breeded our ideas about cool things and how you could do them. How does a submarine work, what/how do you make it (sub)merge, how could you make it autonomous, how do you navigate, how does INS systems work, and so forth. Or whats the cheapest and simplest way to crossbreed a bicycle with a floating device so that you can bike around on water.
Not being able to talk to fellow colleagues about ideas and solutions to problems is probably among the biggest speedbumps in the entire system imho. One of the reasons I believe Google (for example) has been so successful is that the level of trust bestowed on their employees, and the level of openness within the company. People dare talk to eachother without the imminent threat of being fired or scalded for talking about the wrong thing, with the wrong person (this is actually not unheard of, where I am right now). Correct me if I’m wrong, but it’s one of the impressions I have gotten, without ever being a google employee myself.
Finally, don’t get me wrong on this subject. To live in a world, you have to adapt to the world. I’m not about to go divulge ideas and patentable IP on a whim that I hear about while at work. It is a fine line to cross between understanding and sympathizing with an ideology/theory and actually trying to live like an idealist, and it’s not one I would cross at this point of my life.
If/when I do get ideas on my own however, I would most likely not even consider the possibility of patenting it. I’d rather publish it openly so that it could benefit the world rather than try to suck the living bone marrow out of my victims. As a final thought, has anyone considered making an open source/open media “prior arts database” to, which would gather data and ideas on prior arts to patents, hence rendering current/future patents unusable?
embedded Linux seminar 25-27/11 2009
On the 25-27 of november I will be on tour with nohau.se and hold a embedded linux seminar. The entrance is free, but requires a registration, see the embedded linux seminar webpage. according to the following schedule:
- Göteborg 25/11, em 13-16. Plats: Centralhuset Konferens, lokal Orientkusten
- Stockholm 26/11, em 13-16. Plats: Kista Konferens, lokal Alfa
- Oslo 27/11, fm 8-12. Plats: Thon Hotel Vika Atrium, Munkedamsveien 45, Oslo
I will specifically do the Development using Embedded Linux track, which will be 50 minutes long. The presentation is still fairly crude and rough around the edges, but some of the bulletpoints I’m going to talk about is:
- When to use Linux/ When NOT to use Linux
- Pitfalls of open source vs closed source and vice versa
- Hardware vs Development time cost decisions
- Choosing the right hardware
- Choosing the right software
- Security.
I hope to see some of you at the seminars!