Using AWS EC2 instances for large builds

I experimented a few years ago with using EC2 spot instances (virtual server on the internet, but using unused server capacity). It was fairly successful, being able to run large calculations that should have taken weeks in a matter of days.

Since I started at my current job I’ve been running into building increasingly complex yocto images which keeps growing in size, at this point most images I build can take up to 6-7 hours to build on my laptop. This is an i7-4558U 2.8GHz cpu and 8 gigs of RAM so it’s not bad really, just not enough for these types of builds.

Again I started experimenting and I am really happy and impressed. So far all experiments are for open source projects etc, so nothing that has any non-disclosure agreements or corporate etc etc, I’d like to but this isn’t up to me really. I’ve setup an AMI on EC2 which I can instantiate and have up and running in 2-3 minutes, and then mount a 100 gig EBS volume where I store the sources and build data.

The same build that generally takes up to 6 hours on my laptop takes approximately 30-40 minutes on an EC2 c4.4xlarge machine (16 cores and 32 gigs ram).

My key findings so far is:

  1. Keep an AMI with all the build tools necessary/or script the setup.
  2. Keep an EBS volume with the long term stored data, gits etc for building and mount somewhere suitable in the AMI.
  3. Keep a micro instance (these are free/very cheap) around for mounting the EBS volume when you just want to check stuff out, mess around etc but not make builds.

Qt5.5 QtPositioning PositionSource unable to load due to missing symbols

December 31, 2015 by · Leave a Comment
Filed under: Development, Linux, Ubuntu 

I’ve slowly been working on a “Magic Mirror” for a long time now, trying to pick up some new technologies and making something fun. I’ve been working with Qt on and off for the last 10 years or so, but only peripherally, looking and awing over what they do. QML is absolutely awesome in my humble opinion.

A few weeks ago I started using Qt5.5 and ran into some issues executing the Magic mirror in a qmlscene now that I continued the project. It was fairly easy to reproduce but it seems to only affect the Qt binaries I’ve installed from the installer downloaded from qt.io. I’ve had verification that the code works with packages built from source, and trying to verify this on my own as well right now (building as we speak).

This is the sample code not working with qmlscene:

import QtPositioning 5.5
import QtQuick 2.0

Rectangle {
id: root

PositionSource {
id: positionSource
}
}

Bug is reported to qt.io here: https://bugreports.qt.io/browse/QTBUG-50227

Build ppa package

November 26, 2014 by · Leave a Comment
Filed under: Debian, Development, Linux, Ubuntu 

To build a package for ppa distribution, you need some tools. To “cross compile” for releases, for example i386 and amd64 packages on the same machine, takes some more work with schroot, dchroot etc. I’ll start with explaining how to create a “local” package for your own host, I’ll add another entry on how to do an i386 package from amd64. Everything is done on ubuntu 14.04 amd64 machine in this case, and I’m rebuilding dbus.

In short you need:

  1. apt-get install build-essentials dpkg-buildroot schroot gpg
  2. gpg –gen-key
  3. apt-get build-dep dbus
  4. mkdir dbus-amd64 && cd dbus-amd64
  5. apt-get source dbus
  6. export DEB_SIGN_KEYID=
  7. cd dbus-directory
  8. make changes.
  9. dch -i
  10. dpkg-source –commit
  11. dpkg-buildroot -i -I

If you plan on publishing your deb packages to launchpad or some such, you need to create an account and add a ppa. This is simple and done via the http://www.launchpad.net webpage. The webpage also gives you good upload information. Note that they require signed files, so signing must work for you first.

8. Create account on launchpad.
9. Export the gpg generated key to hkp://keyserver.ubuntu.com:11371 (easiest to do via Passwords and Keys tool
10. Import the key to launchpad using the key fingerprint.
11. Create a new PPA from the launchpad dashboard
12. dput ppa: dbus_1.6.18-0ubuntu4.4_source.changes

The package will be built by launchpad on its own, this may take some time..

Saleae16 Logic16 100M 16 Channel Logic Analyzer

June 7, 2014 by · Leave a Comment
Filed under: Development, Hardware, Linux, Robots, Ubuntu 

I got a clone of the Saleae16 Logic16 100M 16 Channel Logic Analyzer a few days ago to test some of the arduino stuff I’ve been working on for the last few months.

I was at first really annoyed because the software simply refused to work, I tried it on 4 different computers, ubuntu 13.10 64bit, Windows 7 32bit and 64 bit and same error for all of them, logic too slow for speed, try reducing blah something. My searches came up with very little info.

So in the end I found the sigrok package and compiled myself (old version in Ubuntu lacked saleae support) and was pleasantly surprised! It works and actually seemed to be an OK package! There are a few things to do before that which wasn’t obvious, you need to extract firmware from the Saleae software suite and so on for example. Also there are a few bugs in the graphical interface that I found so far. I’ll try to get them reported ASAP and possibly do some work on them. I’ll get back to this topic a bit later as it wasn’t completely obvious how to get it running at all points, for now however, it works;).

Ubuntu 12.10 on Dell Precision M4600 APIC issues

February 4, 2013 by · Leave a Comment
Filed under: Hardware, Linux, Ubuntu 

I’ve recently installed Ubuntu 12.10 on a Dell Precision M4600 and had some APIC issues when trying to reboot, the same as everyone else seems to have pretty much. After upgrading the BIOS from A08 to A13 the issues seems to have gone away (not extensively tested, but so far so good).

Before the update, the computer got stuck on the last step of shutting down for restart, but after upgrading to A13 the computer is no longer stuck at that point anymore.

XBMCBuntu stole my heart

November 1, 2012 by · Leave a Comment
Filed under: Linux, Ubuntu 

As some who have read this site before might remember, I didn’t have much luck with LinuxMCE as a media center for my home (basically I couldn’t get any functionality working properly. I got a lot of flack for being pissed and not having talked to the devs/asked for support, which I can understand to some degre).

Anyways, I later on installed XBMCBuntu on my media center that was previously running Mythbuntu (MythTV). I semiliked Mythbuntu, but it had some really nagging issues where it lost my remote control settings and some other settings every time i updated those libraries etc. XBMCBuntu however has really grown on me, some of the functionality is just plain awesome in it, and the ease with which I set it up was amazing. Basically, I had the foundations set up, installed and working within 2-3 hours, downloading subtitles is a breeze, the remote control apps works for my phone and both tablets I have and they got some really nifty functionality (pausing videos when i get a call, scroll through lists of my videos directly in the phone, etc).

XBMC also has excellent support for a ton of different plugins, for example I love the video plugins for SVT Play, TED Talk, and so on. So far, I have nothing but the best to say about XBMC, and I’ve used it for 2-3 months now. The only issues I have are connected to my hardware which is behaving rather strangely (unpredictable boot order and turning on the ethernet wakeup interrupt triggers the machine to boot on its own for some reason. Both issues are unrelated to XBMC however, and the hardware is quite old so…).

Bugs, bugs and more bugs

December 12, 2010 by · Leave a Comment
Filed under: Development, Linux, Projects, Ubuntu 

Lately, I’ve come to realize more and more that bug handling in open source, and specifically in Ubuntu has dramatically declined in efficiency. For years I’ve been extremely satisfied with using Linux because it’s bug free, there has simply not been any serious bugs that I’ve run into. In the last weeks, I’ve run into several more or less serious bugs in Ubuntu, which got me looking at how the bug handling is done.

First off, a few weeks ago, I ran into a bug with Ubuntu 10.10 Ubiquity (the Live CD installer) where I accidentally marked my old /home drive as ext4 when it was ext3 (but not to reformat it). The installer complied happily, and set it up as ext4, but once it got back online, the harddrive was completely wiped. No warning, no nothing. I started looking around, after a while I’ve found several reports on the same matter on launchpad.  For example this and this.

This lead me to take a look at Ubiquity’s other bugs in launchpad, and it’s not very promising. The main installer of Ubuntu 10.10 has 1528 Open bugs as of writing this, of which 846 bugs are new, 35 bugs are marked High importance — and the bugs I found (dare I say, they seem Critical to me, are still not marked with any importance at all). Only 12 bugs are marked as having a patch.

Fine, maybe this is not the poster child of open source. However, the last few days I’ve been severely annoyed by the password popup which is misbehaving. I enter the password, and hit enter (or hit the Authenticate button) and the password field disappears, but the rest of the dialog stays up, and nothing works in it. The only thing you can do is to kill it with the x button. When you do this, you get authenticated…

Since I’m not sure exactly how the authentication is performed in Ubuntu for the update manager etc, I decided to check the update-manager package for Ubuntu on Launchpad. What do I see, if not another package with gigantic mass of bugs filed, but noone dealing with them. 1017 Open bugs, 520 of those are New and 15 marked as High importance. This bug I’ve been having has been reported all over the net, but noone seems to be dealing with it and it isn’t really reported in launchpad. Some computers has it, some doesn’t. It’s nowhere near a critical bug, or even a high importance one, but it’s annoying none the less and it looks extremely crude and comes off giving a fairly unstable feeling.

All this being said, I am wondering how bug handling is done, and how it should be managed on “aggregate” projects such as Debian and Ubuntu. I think the idea is really nice, having upstream bug trackers for each package in the project, but maybe we are spreading too thin having several bug trackers for each minor project? Also, how do we as “normal” users know which package is the reason for the error? I am not so sure it is really the update-manager that is the error in this case, it might as well be some completely other thing behind all that dbus stuff etc. Ie, what is the point of me filing bug reports if I’m not sure they wind up in the right place, or are at all looked after?

Ubuntu 10.10 r8192se_pci driver on the Toshiba Satellite T130-17E (Realtek RTL8191SEvB)

December 10, 2010 by · 4 Comments
Filed under: Development, Linux, Phone, Ubuntu, Windows 

I’ve been home for a few days with a really bad back, and the only thing I’ve been able to do is watch tv, and some minor work with the laptop. I’ve been running Windows 7 which it was delivered with for a few months to get a feel of it, and I must say I was pleasantly surprised for the most part. It hasn’t crashed more than once or twice in three months, and it is fairly snappy (except boottimes seems to get worse and worse, at this point it takes 2-3 minutes to boot). Anyways, for some reason I get a bit sad inside (and bored) every time I boot Windows, there is just “something” about the feel, the look or… I can’t really put my finger on it, that I can’t stand. How the windows open and close perhaps, I just don’t know.

So, yesterday I wanted to test android SDK, re-realized just how much of a bitch it is to install stuff on Windows, so I finally got around to installing Ubuntu 10.10 on it (Already running 10.10 on desktop and mediapc), removing the extra backup partition they deliver laptops with these days. Sidenote, isn’t it a bit like selling a candle with a flammable fire extinguisher to sell a laptop with 500gig harddrive, split it in half, and use on half for “backups”? I digress.

So, installation went almost flawless. The wireless card was identified, saw networks, but was unable to connect to any of them. I managed to pass the installation using trusty old cables, and after installation was done I started fiddling about, reading on the net etc, and found noone who had solved the combination or at least written about it.

Main problem seems to have been hardware wep encoding/decoding, which can be turned off using the hwwep flag to the r8192se_pci module. On Ubuntu 10.10, remove the module, and then reload it by doing this:

rmmod r8192se_pci
modprobe r8192se_pci hwwep=0

If your network works now, you can automate the setting by editing/adding the configuration to modprobe.d, by editing /etc/modprobe.d/realtek.conf and adding the following line:

options r8192se_pci hwwep=0

I hope this has been some help!

Screenmovie 1.1 released

A quick note that Screenmovie 1.1 has been released. It’s still very crude, but adds some sound recording and the ability to turn it on/off. Postprocessing is not supported yet but should be there in the next version.

Features:

  • Record video
  • Record sound
  • Configure file format
  • Configure video codec + settings (5-6 codecs chosen for now)
  • Configure audio codec + settings (2 codecs for now)

I still have some problems, but I just found some info on how to possibly make some problems better at least.

Todo:

  • Fix some high cpu usage problems
  • Add global keybindings
  • Postprocessing encoding
  • Clean up and add some values specific for codecs (as required).

Using Ubuntu as Media Server for Xbox360

November 21, 2009 by · Leave a Comment
Filed under: Personal, Ubuntu, Windows 

I got a Xbox360 since about a year and I just noticed it had some way of connecting to a PC, using the PC as a Media Server. Unfortunately it required a Windows Media Center installation to work, or so it claimed at least. This is probably not news to anyone, but it was very easy to get Ubuntu (or any other Linux distro as a matter of fact) to serve media for the Xbox 360. Xbox 360 uses UPnP to get media from the Windows Media Center PC. To make any recent Ubuntu able to serve UPnP suitable for the Xbox, do the following:

  1. sudo apt-get install ushare
  2. sudo dpkg-reconfigure ushare
  3. sudo vim /etc/ushare.conf
    1. Make sure all the settings are correct.
  4. sudo vim /etc/default/ushare
    1. Make sure it contains USHARE_OPTIONS=”–xbox”.
  5. sudo /etc/init.d/ushare restart

You should now be able to find the PC by searching for it from the Xbox interface (the name you set in ushare.conf should show up in the list of found PC’s). Now that that’s said, I should hint that the Xbox360 has a really shitty availability of audio and video codecs, and I don’t know if it’s possible to resolve this problem. There are hints that there is something called a UPnP Media Adaptor on the ushare website which should be able to convert to proper file formats as necessary, but ushare does not have that ability. Of course, that would give a shitload of cpu load on the fileserver as well, something which sounds less good in my opinion.

My personal opinion so far, Xbox 360 media center is really simple to use, but the available codecs, flexibility and scalability is catastrophically bad in comparison to my MythBuntu installation (still running 9.04 though). The Mythbuntu installation is a bit heavy on the configuration, but much more flexible, handles almost all codecs I’ve run into without even a hitch, and very scalable.

Next Page »