Debugging Linux Kernel Modules with VirtualBox and KGDB

I found a few different pages with information on debugging a Linux kernel with kgdb. There wasn’t a single source with all the information I needed to get set up and working though. So here is how I set things up on my Linux host machine to debug a target Linux kernel running in a VirtualBox VM.

  1. Set up the virtual machine

    When setting up the VM, add a serial port and set it to use Host Pipe. You can check the box to have VirtualBox create the pipe too. Give a path to the file on the host where you want the pipe. This will become the virtual serial console that gdb connects to on the host machine.

    Install VirtualBox with a VM containing the kernel you want to debug. Build the kernel with debug symbols, kgdb support, sysrq keys

    Kernel hacking-> 
       [*] Magic SysRq Key
       [*] Kernel debugging
       [*] Compile the kernel with debug info
       [*] KGDB: kernel debugging with remote gdb ->
          [*] KGDB: use kgdb over the serial console
    

    Build your kernel and install the modules like you would any other kernel.

    Add kgdboc=ttyS0,115200 to the kernel paremeters (ttyS[0-3] depending on which COM port you choose in the VM setup.)

    Copy the vmlinux (uncompressed version to the host machine).
    Also copy the .ko for the module you want to debug.

    Start VM.

  2. Get the target VM ready for debugging

    After your module is inserted, you need its .text address so that you can see the source lines in gdb.

    cat /sys/module/XXX/sections/.text
    0xffffffffa00c0000
    

    When you’re ready to start a debug session:

    echo g > /proc/sysrq-trigger
    
  3. Ready the Host to connect to the VM

  4. You need to convert the pipe that VirtualBox created to a serial port like object. socat does that job:

    > socat -d -d /path/to/pipe pty &
    2010/04/29 12:18:44 socat[4411] N successfully connected from local address AF=1 "\0\0\0\0\0\0\0\0\0\x02\0\0\0\0[\0\0\0|\0\0\0w"
    2010/04/29 12:18:44 socat[4411] N successfully connected via C\xE66
    2010/04/29 12:18:44 socat[4411] N PTY is /dev/pts/3
    2010/04/29 12:18:44 socat[4411] N starting data transfer loop with FDs [3,3] and [4,4]
    

    Notice that your dev is pts/3.

    Now we make a .gdbinit file so you can start gdb without typing the commands every time.

    file vmlinux
    set remotebaud 115200
    target remote /dev/pts/3
    

    You’ll change your pts setting to whatever is relevant.

    start gdb.

    On the target, when you send the g to sysrq-trigger, you should drop to a breakpoint in gdb.

    Next job is to load the module symbols.

    > add-symbol-file  <.text address>
    > b some_file.c:NNN
    > c
    

    Now you can set breakpoints or press c to continue. The target should continue until it comes to a breakpoint or encounters an error.

A last tidbit, when I wanted to close gdb on the host, use detach instead of quit.

Happy debugging!

Adding more disk space with LVM2

I’ve always known that virtualizing things can make management of all types of resources easier. Recently, I had the most pleasant experience adding disk space to a virtual machine. Of course, if you use LVM, this can happen just as easily with real physical disks, but for me, I was able to do this without restarting my machine.

Issue: I’m out of disk space on my root partition.
Solution: The root partition is created on a logical volume with LVM2. Just add another disk, extend the volume group, and then extend the logical volume.

# Added new physical partition /dev/sda3
# create a physical volume out of it
> pvcreate /dev/sda3
# Now, add it to the volume group that my logical volume is on
> vgextend VolGroup00 /dev/sda3
# Now that the volume group has more disk space, the logical volume can grow
> lvextend -L+11G /dev/VolGroup00/LogVol00
# Ok, last of all, I want to filesystem to recognize that more space is available
> fsadm resize /dev/VolGroup00/LogVol00
# sweet, I have more space now
> df -h

All that was done without having to take the system off line. Linux makes life easy sometimes doesn’t it!

An Open Source Library for Writing Firmware on the Cypress FX2 with SDCC

After playing around with the Cypress FX2 and SDCC for the past while, I’ve developed a library of utilities that make some of the common tasks for writing firmware and performing certain functions a little easier. I’m lucky to work for a company that approves of the open source initiative and believes that it is beneficial to give back as well as receive from a wider audience of developers. That being said, I’ve created a git repository with my library:

Here is fx2lib on github: http://github.com/mulicheng/fx2lib/

You can clone it with git like this:

> git clone git://github.com/mulicheng/fx2lib.git fx2lib

Here is a short list of some of the things you can do with this library:

  1. Read/Write data on the i2c bus
  2. Handle USB and GPIF interrupts
  3. Read/Write data on to a serial console
  4. Handle the common USB vendor commands
  5. Implement your own vendor commands
  6. Program the GPIF

Hope you enjoy!

Update 12/15/08: Added project home page at Sourceforge: fx2lib home.

A few open source tips for the Cypress FX2LP (EZ-USB Cy7c68013A)

I’ve had the enjoyable experience of playing around with the Cy7c68013a chip on the Ez-Usb development board for the past few weeks. I thought I’d post a few tips for developing in this type of environment with open source software.

  1. Convert to Linux

    The tools that Cypress provides with the development kit are Windows based. You can download a number of their items directly from their website, but i didn’t find a way to unpack their source code examples without running their installer in a Windows session. The installer creates a new directory: C:\Cypress\ You can ignore the bin directory. Copy the docs directory (You’ll need the TRM) and the examples directory to your Linux machine. The examples won’t compile on Linux (The c51 compiler provided by Keil is Windows based) but they provide good insight into how to do a few things when it isn’t immediately clear by reading the TRM. You may also be interested in their util sources but I’ve found only minor uses for these since there are open source alternatives to a lot of what they do.

  2. Convert to SDCC

    On Linux, you’ll compile programs for the 8051 chip with SDCC. You can install it from their sources or from a package provided by your Linux distribution. SDCC works very similar to gcc but provides output that can be loaded onto embedded devices.

    A couple important switches for compiling:

    • -mmcs51 Tells the compiler to produce output for the 8051 chip
    • -xram-loc Tells the compiler where to start putting xdata variables
      (If a program suddenly doesn’t work after you add an xdata variable, try playing with this 1st.)
  3. Convert the fx2regs.h file

    The fx2regs.h file provided with cypress won’t compile with SDCC (It’s Keil specific). There are a few things you can do.

    • Just use the 8051.h (or 8052.h) file provided by sdcc. (Works if you don’t need lots of the extended registers. Also works for getting started with basic examples.
    • Rewrite it in non compiler specific macros. SDCC comes with a compiler.h file that has macros for SFR, SBIT, etc. You can use these and generate a header file that can be compiled on more than one platform.
    • Use a convert utility. There is a perl script on SDCCs contrib folder that can convert Keil to SDCC headers.
      (For clarification, this script is included in the source tarball.)
    • Rewrite it yourself by hand. Well, if you don’t trust the conversion script anyway.
  4. Learn how to run programs

    The Cy7c68013a chip handles some basic USB commands for you. 1st of all, it will enumerate itself. 2nd, it will allow you to upload and download firmware to the 8051. This 2nd tidbit is vital. You can 1st upload a byte to the reset bit location in firmware (putting the chip in reset), then upload a new firmware, and then upload a byte that puts it back out of reset, thus running your program.

    This fairly straight forward process is documented in the TRM. Even cooler though, someone already implemented it for you so you don’t have to. I use the cycfx2prog utility. It’s GPLed and the source is available to download.

    After compiling a simple program (which produces an ihx-intel hex file), you can upload it like this:

     > cycfx2prog prg:program.ihx run
    
  5. Port a couple example programs to SDCC.

    I started by rewriting a few of the examples and libraries for testing. I recommend 1st getting serial io working so you can debug easier. After that, play around with the i2c board and lights. Finally, test writing to eeprom. (For eeprom images, the hex2bix program that comes with the development kit can easily be ported to run on Linux by using linux time libraries instead of the windows files.)

    Perhaps this section deserves it’s own blog post later on.

Anyway, I haven’t yet found anything that can’t be done on Linux with the Ez-USB kit.

UPDATE 12/5/08: Added Open Source FX2 Library.

Timing C/C++ Code on Linux

For my last post, I played around with C++ and a little programming competition. While on the topic, I decided I’d get slightly more serious and enter the next challenge.

One of the things that slightly annoyed me during the process is having to compile/run the program on Windows to enter the competion, while I’m not opposed to this, I do most of my work during the day on a Linux machine and wanted to be able to use that to develop too. For the most part, this isn’t much of an issue because the code is pretty much standard C++ and is quite cross platform compatible. I just generated a Makefile as well as the Visual Studio Project [1] and I can work on the challenge whenever I feel like devoting a little time.

Anyway, the timer code supplied is not cross platform. It only works on Windows. Since it isn’t that complicated, and I wanted to see times on both platforms, I modified the hr_time.h and hr_time.cpp to include an additional implementation for Linux.

Here are the details:

Linux Time Structures

In <sys/time.h>, you get a few functions and structures that make high resolution timing easy. The gettimeofday function returns the seconds and microseconds since the Epoch. The function operates on the timeval structure:

#include 
// snip...
timeval t;
gettimeofday(&t,NULL); // ignoring the 2nd parameter which is the timezone

In addition to being able to query the time, there are also functions for adding, subtracting, and comparing times. For this purpose, the timesub function works perfectly. You need two values (already queried with gettimeofday and and result timeval:

timeval start,stop,result;
// query the start/stop with gettimeofday during your program
timersub(&start,&stop,&result);

result now contains the difference between start/stop in seconds and microseconds.

For the sake of compatibility, the hr_time implementation wants a double for the time, where the fraction part is the number of microseconds. The nice thing is that timesub normalizes tv_usec to somewhere in the range [0,100000). Here is what I did to convert the result to a double:

return result.tv_sec + result.tv_usec/1000000.0; // 1000000 microseconds per second

Here are the completed code files if you’d like to download and look closer or use for yourself:
hr_time.h
hr_time.cpp

Have fun coding!

[1] Actually, I used Bakefile to generate the project files.

Using Multiple Python Environments With Gentoo

It’s been some time since Python 2.5 became stable and released. Version 2.5 has plenty of new features that have helped me in deciding that it was time to go ahead and start using it for primary development of all my new projects. One of the reasons I was still using version 2.4 is that Gentoo hadn’t upgraded 2.5 to the stable package system.

I decided to go ahead and unmask version 2.5 anyway. Installing Python 2.5 isn’t very complicated and I’ll leave the details out. I’ll just mention that after unmasking Python with the ~x86 keyword and installing the package, you’ll need to run the python-updater. Python-updater had it’s own problems with not being able to find some packages it thought needed to be re-emerged but I found pretty much all of those were unneeded old dependencies and I simply un-emerged them.

When you update Python, you can still get to your old version of Python by tacking on the version number to the python command, e.g., /usr/bin/python2.4. Since the Python updater uses emerge to install your python dependencies in the site-packages of your Python installation and emerge unmerges the old versions, your old Python probably doesn’t have all the site-packages any longer. This is only an issue if you find you need the old Python.

For me, I have a couple applications that didn’t quite want to work with Python 2.5 for some reason. I decided to use VirtualEnv to work on those applications.

The Steps

  1. To install virtualenv, you need setuptools. That package was one of the packages transfered to the 2.5 site-packages install and was no longer available with python2.4.

    To get around the issue, use ez_setup.py to install a 2.4 version of setuptools instead of using emerge.
    cd <working dir>
    wget http://peak.telecommunity.com/dist/ez_setup.py
    /usr/bin/python2.4 ez_setup.py setuptools

  2. I simply used the virtualenv command that came with python2.5 but changed the interpreter to by python2.4 instead of python.

    cp /usr/bin/virtualenv .
    # edit virtualenv to have the correct interpreter line
    vim < or whatever editor > virtualenv
    ------ snip local virtualenv ------
    +#!/usr/bin/python2.4
    -#!/usr/bin/python
    ------ snip ----------------------
    ./virtualenv --no-site-packages <virtual env install dir>
    cd <virtual env install dir>/bin/
    ln -s python2.4 python
    cd <working dir>

  3. Use the old python in it’s virtual environment:

    source <virtual env install dir>/bin/activate
    python
    Python 2.4.4 (#1, Mar 5 2008, 10:47:15)
    [GCC 4.1.2 (Gentoo 4.1.2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

Anyway, that’s it. You can have a virtual environment that uses any version of Python you like on your system. I imagine this procedure would be somewhat similar on pretty much any Linux Distribution.