Playing with bcache: NVMe Write-Back Caching over HDD

Over the weekend I decided to revisit bcache, Linux’s block-layer cache system, to try and squeeze more performance out of a large spinning disk by caching it with a small SSD. The concept is simple: attach a fast SSD (in this case, a 256?GB NVMe drive) in front of a slower HDD, and let bcache absorb and smooth out writes using the SSD as a write-back cache.

Unfortunately, the results were not what I expected. Instead of a big performance bump, I discovered that my NVMe drive—despite being a Samsung MZVLQ256HBJD—is shockingly slow for sustained writes, barely better than the HDD I was trying to accelerate.

Setup Steps

Here’s the setup I used:

1. Wipe the drives
   Make sure there’s no lingering filesystem or bcache metadata:

   sudo wipefs -a /dev/sda
   sudo wipefs -a /dev/nvme0n1p4

2. Create backing device on the HDD

   sudo make-bcache --wipe-bcache --block /dev/sda

3. Create cache device on the NVMe SSD

   sudo make-bcache --cache --wipe-bcache --block /dev/nvme0n1p4

4. Attach the cache to the backing device
   Get the cache set UUID and attach it to the backing device:

   # Find UUID
   bcache-super-show /dev/nvme0n1p4 | grep UUID
   
   # Attach (replace UUID)
   echo | sudo tee /sys/block/bcache0/bcache/attach
   

5. Enable write-back mode

   echo writeback | sudo tee /sys/block/bcache0/bcache/cache_mode
   

6. Format and mount

   sudo mkfs.ext4 /dev/bcache0
   sudo mount /dev/bcache0 /mnt
   

Benchmarking Reality

To get a baseline, I ran a basic fio test directly on a file on my root filesystem, which resides on the NVMe drive:

fio --name=test --filename=/home/osan/testfile --size=10G --bs=1M --rw=write \
--ioengine=libaio --iodepth=32 --direct=0

The result: ~194?MiB/s write bandwidth. And that’s not a typo.

With clat latencies in the hundreds of milliseconds and 99th percentile latencies >2?seconds, the drive showed behavior much closer to a spinning disk than what you’d expect from an NVMe device. Even enabling write-back caching in bcache didn’t improve performance much—it simply matched the NVMe’s raw write speed, which isn’t saying much.

The disk shows 8GT/s x4 on PCIe Gen3, so the bus isn’t the bottleneck. The drive is just bad at sustained writes—probably DRAM-less and overaggressively optimized for client workloads or bursty activity.

<H2Takeaways

If you’re planning to use an NVMe SSD as a bcache write-back device, don’t assume all NVMe drives are fast. Many low-end OEM drives—especially DRAM-less models like the MZVLQ256HBJD—fall off a performance cliff under sustained write load.

In my case, bcache didn’t offer a performance boost because the cache layer was just as slow as the backing device. In retrospect, I would’ve been better off checking the SSD’s sustained write behavior first using something like:

sudo nvme smart-log /dev/nvme0n1

Or running long-duration writes and monitoring latency and thermal throttling.

There’s still value in bcache—especially if you have a real SSD or enterprise-grade NVMe to use as a cache. But with a sluggish consumer NVMe like mine, bcache turns into a no-op at best and a source of extra latency at worst.

Iptables tutorial 1.2.3

It’s time for the… 4 year update?…

I recently got a lot of very good feedback on the iptables tutorial by Ed Drouillard. It’s been laying dormant for a long time, but the feedback kind of got me reminiscing about writing tech documentation etc, and I decided to get some work done on the old documentation.

My immediate concern is that the build systems etc are kind of not working, and we’ve identified some technical issues with the actual documentation and rectified it.

We’ve also had to more or less reproduce/fix the build procedures as it’s been ages since I even tried building the project. I’m surprised at how resilient it has been though, it mostly boiled down to installations of packages and changes in settings etc.

For now, you’ll find the updated version in Iptables-tutorial.

Iptables-tutorial work

It’s been a while since I looked at the iptables-tutorial to say the least. The last real commits where 9 years ago and the last proper release is closer to 13 years ago….

The last two three days I kind of picked it up again, for fun mostly. In the end of my maintenance of the project I kind of burnt myself out on the whole topic, I just did not want to do the whole thing anymore. I’ve grown and changed as a person since then. I don’t have the same spare time for one.

My first order of business, The build system was always a mess and I started cleaning out stuff that shouldn’t be there. A bunch of old scripts have been removed, I managed to remove the dependency on the fabpdf backend for jade and also the eps_to_png script with that, and almost all of the changes.sh scripts where removed. The Spanish and Portuguese builds where similarly cleaned up. Finally a Travis file was added to get automated builds running on Travis, and this actually works now!

I’m getting close to making a 1.2.3 release imho to get something new out there. The actual content has barely changed to be honest, maybe a few words at the most, but it feels like something that would be nice to get out there.

The task of getting this documentation up to par is a tremendous effort to be honest, and I’d be really interested in getting help from anyone who reads this. If you feel like contributing, contact me, check the code out on github, add bugs/tasks on stuff you find that is wrong, or provide pull requests. I would be thrilled to have other people working on this as well so it becomes more vibrant again and don’t stagnate as it has done over time.

Qt5.5 QtPositioning PositionSource unable to load due to missing symbols

I’ve slowly been working on a “Magic Mirror” for a long time now, trying to pick up some new technologies and making something fun. I’ve been working with Qt on and off for the last 10 years or so, but only peripherally, looking and awing over what they do. QML is absolutely awesome in my humble opinion.

A few weeks ago I started using Qt5.5 and ran into some issues executing the Magic mirror in a qmlscene now that I continued the project. It was fairly easy to reproduce but it seems to only affect the Qt binaries I’ve installed from the installer downloaded from qt.io. I’ve had verification that the code works with packages built from source, and trying to verify this on my own as well right now (building as we speak).

This is the sample code not working with qmlscene:

import QtPositioning 5.5
import QtQuick 2.0

Rectangle {
id: root

PositionSource {
id: positionSource
}
}

Bug is reported to qt.io here: https://bugreports.qt.io/browse/QTBUG-50227

Iptables-tutorial and ipsysctl-tutorial on github

I guess I should have done this a long long time ago. Both the iptables-tutorial and the ipsysctl-tutorial source code are now available on github. Many many years ago I had an idea of putting the version control system out there for use, but I never realized it for various reasons. Both these documents are old by today, but the basic information is still valid and will likely be for a long time to come it seems.

I apologize for the version history, I moved from CVS in a rather rude way to SVN without keeping the history, which was what I used back in those days.

I invite anyone and everyone to do edits if they wish to and send me pull requests to fix the issues they find, or to add the documentation they’d like to add.

The iptables tutorial is available at:
https://github.com/frznlogic/iptables-tutorial

The ipsysctl tutorial is available at:
https://github.com/frznlogic/ipsysctl-tutorial

Build ppa package

To build a package for ppa distribution, you need some tools. To “cross compile” for releases, for example i386 and amd64 packages on the same machine, takes some more work with schroot, dchroot etc. I’ll start with explaining how to create a “local” package for your own host, I’ll add another entry on how to do an i386 package from amd64. Everything is done on ubuntu 14.04 amd64 machine in this case, and I’m rebuilding dbus.

In short you need:

1. apt-get install build-essentials dpkg-buildroot schroot gpg
2. gpg –gen-key
3. apt-get build-dep dbus
4. mkdir dbus-amd64 && cd dbus-amd64
5. apt-get source dbus
6. export DEB_SIGN_KEYID=
7. cd dbus-directory
8. make changes.
9. dch -i
10. dpkg-source –commit
11. dpkg-buildroot -i -I

If you plan on publishing your deb packages to launchpad or some such, you need to create an account and add a ppa. This is simple and done via the http://www.launchpad.net webpage. The webpage also gives you good upload information. Note that they require signed files, so signing must work for you first.

8. Create account on launchpad.
9. Export the gpg generated key to hkp://keyserver.ubuntu.com:11371 (easiest to do via Passwords and Keys tool
10. Import the key to launchpad using the key fingerprint.
11. Create a new PPA from the launchpad dashboard
12. dput ppa: dbus_1.6.18-0ubuntu4.4_source.changes

The package will be built by launchpad on its own, this may take some time..

DBus remote connection

In a project I’m working on at the moment we wanted to remotely monitor a DBus session bus. The system in question has several buses available using different users and systemd. d-feet can connect and monitor remotely via TCP.  The following code will allow you to connect to a remote DBus on a target development board for example.

First copy the original session.conf to separate configuration files for each user.

cp /etc/dbus-1/session.conf /etc/dbus-1/session.conf.<username>
cp /etc/dbus-1/session.conf /etc/dbus-1/session.conf.<username2>

Then for each of the newly created configuration file, add the following configuration but with different port numbers and correct username director in /run/user. The ip address should be the IP of the connecting host, not the server. Edit session.conf.<username> and add:

<listen>tcp:host=<ip>,bind=*,port=<port>,family=ipv4</listen>
<listen>unix:path=/run/user/<username>/dbus/user_bus_socket</listen>
<listen>unix:tmpdir=/tmp</listen>

<auth>ANONYMOUS</auth>
<allow_anonymous/>

The systemd script is rewritten to use a specific conf file for the specific user trying to start the DBus.

Edit  /lib/systemd/system/dbus-session@.service and rewrite the ExecStart line as follows.

ExecStart=/usr/bin/dbus-daemon --config-file /etc/dbus-1/session.conf.%i --nofork

This allows you to connect using d-feet or other dbus applications (potentially, you should be able to connect for example other services over the network to the new DBus….).

Choose “connect to other bus” and use as bus address:

tcp:host=<targetIp>,port=<port>

Done. Hopefully.

Testing out RasPlex

I decided to try out RasPlex after having run OpenElec and RaspBMC for about half a year, and before that XBMCbuntu for a few years. I’ve been using plex server on my file server for a few months already and plex on the tablets/phones to stream movies, tv shows, etc while in bed. I haven’t had that much time with RasPlex yet, but I’m really impressed so far, especially when you start looking at what type of infrastructure it enables. I’m now using the same chain of applications for my tv, phones, tablets, and laptops, this is something I’ve never been able to before (if you except NFS/SMB solutions with VLC or some such video player, which has proven unstable for the phone/tablet cases in my case, high def videos will stutter a lot among other things).

My main problem with the entire setup is that my fileserver is a bit too weak to transcode 1080p on the fly, it is a D525 Atom CPU however, and I didn’t see this usecase when I bought it, so I’ll have to live with it for now ;).

All that said, I’m still very happy with XBMC, but I feel that RasPlex so far has given a more… mature feeling. It seems to work very well with no big hickups so far, the UI on RaspBMC and OpenElec was really slow, something that still holds true in RasPlex, but it is better.

Raspberry Pi + 2x Arduino Nano V3.0

Quick update, during my evenings I’ve been working with one of the Raspberry Pi’s I won on a local contest a few months ago, and it’s generally derailed into some kind of “let’s put as much stuff on it as possible”, meaning that I currently got my Raspberry Pi hooked up with:

• Slice of Pi
• Adafruit PWM driver
• Raspicam on a simple pitch/yaw servo gimbal that me and my 1,5 year old put together in 10 minutes. Controlled via PWM.
• MPU9150 sparkfun breakout board
• 2 Arduino Nano V3.0

The two Arduino Nanos have split functionality, where one will provide me with data that needs to be gathered frequently, and the other is used for slow processes such as reading out 1-wire temp sensors etc.

The first nano will have the following functions hooked up to it:

• 3x HC-SR04 ultrasound distance sensors
• Voltage measurement 0-20V circuit
• Control of 2 relays
• 3x line tracking sensors
• Reed switch
• 2x motor controls via L298P chip

The second nano has the following hooked up to it:

• MPX2200GP pressure sensor (will use something else soon’ish)
• 2x 1-wire DS18B20 temperature sensors.
• Others?

The general idea was to move timing critical stuff off the raspberry to the Nano and let the first one deal with quick sensor readouts, while the second Nano is dropped off with relatively slow sensors (DS18B20 takes very long time to read out for example). The two nanos will talk to the Raspberry via SPI I think, or possibly serial ports, but this is less likely as it would either require me to use one USB serial driver and the raspberry UART or get a 2 port USB hub of some kind and talk via USB UART’s.

I’ve meanwhile also played around with Eagle CAD for the first time in my life, making some electrical drawings of the hookups. I’m considering making a PCB layout of everything when I get there, not sure if there is any interest in this out there except for “the lols”. The image is still very raw and missing a lot of stuff that I know needs to be added at some point.

During christmas I spent some time making opencv haar cascade training on clementine detection and generally fooling around with it. I think I’m leaning towards making a robot (chassi on the way) which will travel around in the room looking for objects… I guess some of the sensors are a little overboard for this, but it’s all fun and games while it’s not work… 😉

Raspicam and OpenCV instructions

I have previously gotten the opencv and python bindings to work via the 2.3 opencv system and facial recognition did work, but the system is bugged out and I could only get 64x64px image size.

I followed these instructions to get the raspicam to work with opencv and simplecv

http://tothinkornottothink.com/post/59305587476/raspberry-pi-simplecv-opencv-raspicam-csi-camera

However, there where some minor details wrong with it so here is a short list of the updates to the installation instructions on that page.

I noticed a few problems as I saw it with the above instructions, or possibly I didn’t read the instructions well enough. That said, very good info! Thank you!

Here are the issues I had:

The opencv_2.4.5.sh script pulls in libopencv-dev which in turn pulls in the entire libopencv2.3 from raspbian repositories, which in turn meant that simplecv was still using libopencv2.3.

apt-get remove libopencv-core2.3

Once that is removed, python-opencv is also removed and simplecv will not even run anymore. Adding the following will add the path to where the python cv bindings are.

export PYTHONPATH=/usr/local/lib/python2.7/site-packages/

And finally, the LD_PRELOAD to actually use the correct uv4l libraries.

export LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so

Once this is done, you should no longer get the munmap errors etc, and large resolution should work:

SimpleCV:1> cam = Camera(0, {“width”:640, “height”:480})

SimpleCV:2> cam.live()

etc.

Next Page »

• Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

• Categories

  • 3D printing
  • Android
  • Communications
  • Configuration Management
  • Debian
  • Development
  • Economy
  • Editors
  • FPV
  • Frozentux.net
  • General
  • Hardware
  • Ipsysctl
  • Iptables
  • Linux
  • Management
  • Netfilter
  • Personal
  • Phone
  • Plastic modelling
  • Projects
  • Robots
  • Ubuntu
  • Uncategorized
  • Video
  • Windows

• Recent Posts

  • Playing with bcache: NVMe Write-Back Caching over HDD
  • Iptables tutorial 1.2.3
  • Sparse files
  • Gitea and Drone.io
  • DevOps World Nice 2018

• Tags

  apt-get arduino c++ C/C++ DBus Debian Dpkg dpkg-buildroot eclipse embedded FFmpeg filesystem FPV github Guruplug hack HTML HTPC iMON iptables iptables tutorial JavaScript jenkins KC910 launchpad LG Linux lirc mirabox opencv php presentations pygtk python raspberry pi Remote Control server servo driver subversion sync trac Ubuntu USB XBMC yocto