Work? What work?

January 27, 2009 by · 1 Comment
Filed under: Configuration Management, Linux 

So, just a brief update. I’ve recently (a few months back *cough*) taken over our Linux “education” group at work, and it’s interesting. The sad part is, we mostly only see people who already knows what Linux is as we’re working internally in a world where most people are rather Computer savvy as it is. It’s given me a few new viewing angles though, and I’ll get back to that at a later point.

Currently working on some Trac guidelines for our Change Management process as well. Working from home today to actually get something done with it, as most of the days I wind up getting too many disturbing calls, talks and discussions to be very efficient. Our first two tries at making a decent workflow winded up a bit messy, and I think we really must get this down properly this time.

There are some other things I react on, and want to fix, for example, as it looks now, every single project sets up their own bugtracking/ticketsystem, and every project uses a different system (trac, mantis, clearcase, dimensions, etc). Preferably, this should be centralized in some fashion, and if possible I’d love to get a bit more homogenized environment. As it is, I try to tell people “look, here’s a system for handling your day to day tasks, use it!”. First time, the workflow got overly complex, second shot was also overly complex, and people where put off by all the choices and steps to take. This problem mainly stems from project/change management criterion.

My latest and greatest (yeah yeah) workflow should alleviate some of these problems by making some of the choices less visible to normal users. Ie, we have one task management system and a problem and change management system baked into one, but normal users (programmers) only use the task management system, while the project manager, tech project manager and CM also have the ability to handle problems and changes in separate workflows.

We’re also adding the ability to have supertickets, where a single problem report can contain several tasks. This is a pseudo development so far, as we’re not actually adding the whole deal right now, we’re just adding the idea of it, not bounds checking or views/reports of it. Basically, every ticket can have a superticket (we add a numeric field to the ticket), which can point to another ticket, which is the “parent” ticket. This makes it possible to handle a large and complex bug in several smaller tickets. Anyways, the idea is there, but it’s not fully implemented. If our management likes it, and the others like it, we could implement it for future usage. I’m worried it’s too complex however. At the same time, one complex system might be better than 6 alltogether different systems as it allows for longer time to learn? Kind of like… well, unix for example. Once you find ls, its a darn good bit faster than having to click your way through a whole heap of paths to find the specified file list.

At the same time, both me and PM are a bit tired of Trac’s shortcomings, maybe change to Mantis for example? My general thought to this however is, we need to stick it out i’m afraid… one more system will just make the normal user less interested in the new tool and hence taking even longer to learn. As it is, people use it at a bare minimum cause they dont know it, give them time to learn it properly, and they might come to like it. Comments on this line of thinking?

For now, tata. Back to writing.

Build components

November 23, 2008 by · Leave a Comment
Filed under: Configuration Management, Development 

After a weekend of work, I finally got myself a build component that I’m semi-pleased with, for C and C++ projects, using Subversion. Most likely works for any other lower level programming language as well.

First off, structure. Each component is it’s own BTT(Branches, Tags, Trunk)-root, residing in a Project_Modules directory in subversion. Each component contains an inc, src, test and a stubs directory. Rationale for the BTT-root is that, with a separate BTT-root for each component we can raise the version of each separate component without having to raise it for the entire project.

The Project directory resides on the same level as Project_Modules, and is empty, only containing the subversion property externals pointing to the trunks of the Components in Project_Modules. Rationale for this is to have a simple place to checkout the entire project. It’s a bit dangerous when working with branches, and requires a little bit extra care so one doesnt write into the trunk out of mistake. Possibly block everyone but a specific user to write in the trunks and have that CM person do all the branching/merging. It is time consuming however.

It looks something like this:

  • Project_Modules
    • Component1
      • inc
      • src
      • test
      • stubs
    • Component2
      • inc
      • src
      • test
      • stubs
  • Project

Inc directory is the public interface of the component towards the other components. Src directory contains the actual code of the component. Test contains unit tests (personally, i create a new directory for each new unit test file). Stubs contains the stubs of my own component. Ie, Component1/stubs will contain stubs for the functions in Component1. Rationale being that 95% of the time, we want to stub another component in the same way, instead of keeping stubs of a component in 10 different components, we keep it in one place.

“New” subversion structure using svn:externals

Me and the boss deviced a new structure for the project during the last few weeks, and it’s been slowly refining in our heads until yesterday when we finally implemented it. I think we made a rather refined and complex structure, but once we got it into place physically and once we get the general idea into the developers heads (including me), I think it will prove very powerful.

That being said, I don’t think this is a new structure, I just think people are very quiet about how they use subversion, and it’s a problem. Newcomers do the same old errors over and over again. So, let’s get on to try and explain it all.

Most projects uses a single BTT root, where BTT stands for Branches, Tags and Trunk. Ie, they start a project, and then straight in the root put the BTT, and then inside that, they create the project structure. For example:

  • project-root/
    • Branches
    • Tags
    • Trunk
      • admin
      • src
      • out
      • test

This is a good basic structure for very small projects, containing perhaps 10’ish files, or where the actual implementation is perfectly homogenous and has no need for separated versioning. Every time we want to make a release, we cheap copy the content to Tags as a new tag (called perhaps /Tags/Milestone1-RC1). We now have a release that we can provide to people.

The problem comes if it isn’t so homogenous. For example, let’s say you are developing a calculator. It has two objects, a numpad and a display. What if you want to make a new version just of the display? You need to make a completely new version, including for the numpad.
Or how about wanting to branch just a small part of the project? Ie, I want to use a branch for the numpad, and then use the trunk for the display. You’d then have to make a cheap copy for the entire tree. Admittedly, it isn’t costing too much.

Our “new” structure deals with this on a different level. Basically, the idea is to have multiple BTT roots, and then use svn:externals to connect the correct tags to create
1) a complete releasable project and
2) a complete workarea project.

For the calculator example, you get the following structure:

  • calculator/
    • Calculator_Modules/
      • Display/
        • Branches/
        • Tags/
        • Trunk/
      • Numpad/
        • Branches/
        • Tags/
        • Trunk/
    • Calculator/
      • Branches/
      • Tags/
      • Trunk/

As you can see, it looks much more complex, and it is, but the possibilities are infinitely much better.

The Calculator/Trunk/ directory contains a svn:externals property linking in the Calculator_Modules/Display/Trunk as Display and Calculator_Modules/Numpad/Trunk as Numpad. This works by linking external resources into the current directory structure, so basically I would get the trunks into my Calculator trunk, but properly renamed, without them actually being there in the repository. This also works on “real externals” by the way, such as linking in a specific version of a library from some repository on the Internet.

To create a Calculator/Tags/MS1 we could either just set a -rXX to the correct subversion revision, or we would create svn:externals to the correct Display and Numpad Tags, not their trunk. This way, we can say that “Calculator 1.0 contains Display 2.0 and Numpad 2.1”, not “Calculator 1.0 contains Display revision 439 and Numpad revision 587”, or even worse “Calculator 1.0 is revision 587” which completely lacks granularity.

I’m not completely sure it’s perfect, and others have probably already tested it, but I think it will be pretty sweet :-).

Dangling locks in the Subversion tree

September 26, 2008 by · 2 Comments
Filed under: Configuration Management 

I recently ran into another rather peculiar and interesting case with subversion. As I’ve already explained, I don’t have actual physical access to the subversion servers we are perusing for our project. We use TortoiseSVN for access to the repository on our local systems.

My boss was working with some excel and word files checked into the repository. He had template.xls locked as he was working in it, he decided to rename it to a template, so template.xlt it is (and checked in). After doing some more work, he decided this was a bad idea, so he changes it back to template.xls, and tries to check it back in again… no luck, it wont work.

15-20 minutes worth of investigation later, we found the problem. TortoiseSVN had been configured to always check “Keep Locks” on commit, hence the client kept the lock when he commited the template.xls deletion. To fix this problem, run svnadmin lslock and then svnadmin rmlock the file in question. Please note that this requires actual access to the physical subversion repository. Http or svnserve will not do in other words.

A step by step on how to cause the Dangling lock problem — All of the actions are done via TortoiseSVN interface:

  1. Create a repository.
  2. Check out the repository.
  3. Add a file to the repository, set the svn:needs-lock property.
  4. Commit the file.
  5. Lock the file.
  6. Change name of the file, and check the “Keep locks” checkbutton.
  7. Commit the changes.
  8. Change the filename back to the original filename.
  9. Commit the change. This step will fail.

This is a rather funny problem imho :-). We finally managed to get it fixed after contacting the IT department and sending them the command. 10 minutes to cause the problem, 20 minutes to analyze it, 20 hours to wait for the fix. Sometimes there should be less depth in organizations.

Svnsync automatisation via Windows Scheduler

September 20, 2008 by · 1 Comment
Filed under: Configuration Management, Development, Windows 

Recently I’ve been setting up for a project, as you might have noticed, and one of the problems we ran into was the fact that we have a centralized subversion repository. We don’t have access to control or install anything on the actual server unfortunately, so hence we can’t install trac on that system, as previously explained. The solution is to sync the main subversion databases to the trac server as previously explained.

A secondary problem arising from this, is that we can’t actually change the hook files of the master repository. The solution to this, is to add a synchronizing scheduler. As I was stressed, and had no time to deal with this properly, I had to run with the Windows Scheduler of all horrible solutions. The following explains how to set it up.

Windows Scheduled synchronize Task

Warning! The windows scheduler does not start automatically when you reboot etc. If you do this, the task restarts at it’s start time, so if it’s set to run at 00.00 every day, and then every 5 minutes, it will restart and start running at 00.00, not as soon as the machine is started up again. This is pure evil.

A scheduled task is set up as follows, create a bat file (ie, c:projectssvnprojectsvnsync-project.bat):

svnsync synchronize http://localhost/svn/project –sync-username slaveuser –sync-password tjohej –source-password password
c:Python25Scriptstrac-admin.exe c:projectstracproject resync

The Second line unfortunately has to be done to resync the new changes with the trac database. After this, create a scheduled task in windows as follows:

  1. Go to start -> control panel -> scheduled tasks
  2. Click Add scheduled task
  3. Click Next
  4. Click Browse…
  5. Browse to your svnsync-project.bat file, as described above. Double click it.
  6. Set name (default is ok). Perform this task: Daily
  7. Click Next
  8. Start time: 00:00, Every day, Start date: today
  9. Fill in username/password to run it as.
  10. Select open advanced properties for this task. Click Finish.
  11. Go to tab Schedule. Choose Advanced.
  12. Click Repeat task, fill in Every 10 minutes, and click Time and fill in 23:59
  13. Click OK.
  14. Click OK
  15. Done.

This could most likely be done from cruisecontrol, but as stated, i’m stressed. I hate windows by now (at/cron is just sooo much nicer to handle).

Trac on separate server from subversion

September 11, 2008 by · Leave a Comment
Filed under: Configuration Management, Windows 

It is possible to have Trac on a different server than the subversion server, albeit a bit convoluted. This is slightly stolen note from http://blogs.open.collab.net/svn/2007/08/mirroring-repos.html

Set up the master subversion server as you would normally. We will get back to this one.

On the trac server, install subversion etc as well. This server will rely on svnsync to become a secondary/mirror subversion server. It must not be used for checking in or working against, it is purely a repository for reading.

Set up trac and all other tools required on the mirror server.

Now, you need to set up user accounts on the mirror server. To start with, create the mirror subversion repository, it will be empty to begin with:

svnadmin create MIRROR_REPOS_PATH

After that, create authz authentication files for the repositories to stop anyone from writing/working against the mirror repository, such as the following:

[/]
* = r
svnsync = rw

You could extend this by writing hook scripts blocking anything but the svnsync user.

If the master subversion repository is live while adding the trac database, all of the following must be done during server maintenance window (ie, close it down from any kind of access, except the mirror machine).

Once all that is done, its time to initialize the the mirror server with the master subversion repository (ie, move over the data).

svnsync initialize URL_TO_MIRROR_REPO URL_TO_MASTER_REPO –username=svnsync –password=svnsyncpassword

And once that has successfully been done, do as follows:

svnsync synchronize URL_TO_MIRROR_REPO –username=svnsync –password=svnsyncpassword

The final step is to either add a scheduler (cron.d) script, or a post-commit hook script running the following:

# Example for synchronizing one repository from the post-commit hook
#!/bin/sh
SVNSYNC=/usr/local/bin/svnsync
$SVNSYNC synchronize URL_TO_MIRROR_REPO –username=svnsync –password=svnsyncpassword &
exit 0

Preferably run it from post-commit as it gives better refresh rate.

You can now run the trac against the mirrored subversion server.

Trac on windows

September 10, 2008 by · 1 Comment
Filed under: Configuration Management, Windows 

Trac is a rightfull bitch to install on win32 as it requires very specific version matching of packages. This is a long winded installation note with the more or less latest versions available as of this writing.

I’m hoping to get working on TortoiseSVN and Trac integration once this is done (ie, make tortoise automatically set variables etc sent in change notes, which can hence update the issue tracking systems in trac and so forth. The way of working is really really sweet imho, and I think it could be a really nice way of working. I’m just sad that I have to set all this serverstuff up on windows though.

Let’s begin with a list of all the installation files used:

python-2.5.2.msi
setuptools-0.6c8.win32-py2.5.exe
py25-pysvn-svn150-1.6.0-975.exe
svn-python-1.5.2.win32-py2.5.exe
Genshi-0.5.win32-py2.5.exe
Trac-0.11.win32.exe

WampServer2.0c.exe
CruiseControl-2.7.3.exe
TortoiseSVN-1.5.3.13783-win32-svn-1.5.2.msi

WampServer2-APACHE229.exe
svn-win32-1.5.2.zip
mod_python-3.3.1.win32-py2.5-Apache2.2.exe

These packages should be fairly simple to get started with. Install them straight on, in the order mentioned.

python-2.5.2.msi
setuptools-0.6c8.win32-py2.5.exe
py25-pysvn-svn150-1.6.0-975.exe
svn-python-1.5.2.win32-py2.5.exe
Genshi-0.5.win32-py2.5.exe
Trac-0.11.win32.exe
Install the above packages in that order. Some errors i ran into:

Unsupported version control system “svn”

I accidentally used svn-python-1.5.0.win32-py2.5.exe. It also complained about not finding SSLEAY32.DLL which threw me off course, looking for the wrong solution.
Secondary applications
On top of this, the following applications where installed (not yet configured/set up).
WampServer2.0c.exe
CruiseControl-2.7.3.exe
TortoiseSVN-1.5.3.13783-win32-svn-1.5.2.msi

WampServer2-APACHE229.exe
The Apache2.2.9 must be installed before svn-win32-1.5.2.zip files are, and the .so files must be put in 2.2.9 as that package will not run on Apache2.2.8. Apache 2.2.8 with svn-win32-1.5.2.zip will die silently without hint as to why it died.
Once all of the above is installed, create a svn repository in c:\projects\svn\test (create directories, and then right click test and choose TortoiseSVN -> Create repository here.
Now create a trac database in c:\projects\trac\test\ by running
c:\python2.5\trac-admin.exe c:\projects\trac\test\ initenv

Answer the questions asked by trac-admin.exe. To test the trac at this point, run tracd.exe per ordinations from trac-admin.exe.
svn-win32-1.5.2.zip
Contains required files to make dav_svn work in apache2.2, edited extract on how to install:
For an Apache server here’s the essentials:
1. Copy bin/mod_dav_svn.so and bin/mod_authz_svn.so to the Apache modules directory.
2. Add the Subversion/bin directory to the SYSTEM PATH.
3. Edit the Apache configuration file (httpd.conf) and make the following changes:
3a. Uncomment the following two lines:

#LoadModule dav_fs_module modules/mod_dav_fs.so
#LoadModule dav_module modules/mod_dav.so
3b. Add the following two lines to the end of the LoadModule section:
LoadModule dav_svn_module modules/mod_dav_svn.so
LoadModule authz_svn_module modules/mod_authz_svn.so
3c. Add the following to end of the file.
<location /svn>
DAV svn
SVNPath c:\projects\svn\test
AuthType Basic
AuthName “My Subversion repository”
AuthUserFile “c:\projects\svn\test\conf\passwd” Require valid-user
</location>
Add users to passwd file above:
C:\wampbin\apache\apache2.2.9\bin\htpasswd.exe -b \projects\svn\test\conf\passwd myuser hejhej

Finally installing Trac in apache2.2

mod_python-3.3.1.win32-py2.5-Apache2.2.exe
To run trac via apache, you need the above module for apache as well. Once this is done, add the following to modules section of apache:
LoadModule python_module modules/mod_python.so

To test the python installation add the following to the end of your httpd.conf:

<location /mpinfo>

SetHandler mod_python
PythonInterpreter main_interpreter
PythonHandler mod_python.testhandler
</location>

And if that works, test the following:

<Location /trac>

SetHandler mod_python
PythonInterpreter main_interpreter
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv “c:/projects/trac/test/”
PythonOption TracUriRoot /trac
</Location>

Further reading
http://tortoisesvn.tigris.org/svn/tortoisesvn/trunk/contrib/issue-tracker-plugins/
http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-propertypage.html
http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-bugtracker.html
http://trac.edgewall.org/wiki/TortoiseSvn
http://trac.edgewall.org/wiki/TracOnWindows
http://ifdefined.com/blog/post/2007/10/BugTrackerNET-integration-with-Subversion—design-choices.aspx

Slow grasswidow evening

What do you do when you are home alone a saturday evening? Personally, I web2.0’d my life (i got facebook, a blog, remember the milk, dipity, evernote, and decided to rewrite/update my webpage). Also, I really got started building the Fokker DR.1.

To update frozentux.net, I screwed around a bit with eclipse pdt. I want to do this since I think the whole site looks pretty awful. I was ok with it “back then”, but not any more. I’ve been using eclipse for a few months to code c, c++, qt and I’ve grown rather fond of it. My first time with eclipse left me… well, let’s just say i went back for a few years to vim and loved it. Still wish there was a decent vim/eclipse plugin as i love vims command/edit mode.

I digress. Basically, what I did was Install subversion, make a copy of frozentux.net in subersion and a few other projects in there. I setup subversion with apache2. I know I suck that haven’t done this before. After that, I installed pdt, subclipse and a few other modules in eclipse and got it all working together. I think I might like the subclipse module, but still got a few things I need to find out.

I’m gonna get started on rewriting frozentux today I hope. Now, going for a walk first.