Eudyptula challenge

Av Marcus Folkesson den 30 Juli 2015

For those who never heard about the Eudyptula Challenge, it is a series of programming tasks for the Linux kernel. The format of these exercises is inspired from the crypto-challenges that has been around for a while.

Before we go any further we need to sort out one thing that I know you are thinking about, why chose a name that is so hard to spell to and impossible to pronounce as Eudyptula?
When signing up for this challenge, all communication is done by mail with Little, who is ”a set of convoluted shell scripts that are slow to anger and impossible to debug”. Eudyptula is latin for the smallest species of penguins, and there is the connection.

To join the challenge, all you need is to send an email to Little and tell that you want to participate. This requires that you have a sane email-client, which surprisingly few have. Nasty things such as HTML in emails are totally forbidden, noone wants it anyway, and it will be rejected from all kernel mailing lists.
I personally use Mutt for all my emailing.

There are currently 20 tasks starts from a very basic ”Hello world” kernel module and moving up in complexity. A couple of tasks is to get patches accepted to the main Linux kernel source tree which involves some ”real work”, interact with people on the mailing list and get to know the development procedures for the Linux kernel.

But what do you need to know to take the challenge?
A basic understanding of the C programming language is required, that’s all.

I have been doing Linux kernel development for quite some time but I have learnt a lot. Mostly because I have not had reason to touch areas such as the implementation of the FAT filesystem, get my modules to load automatically when I plug in an USB keyboard or write a netfilter hook to search all incoming TCPv4 packets for a specific ID.
As you see, the tasks is touching many parts of the Linux kernel. The tasks also cover really useful parts such as using DebugFS, create sysfs entries, make character devices and so on.

One side effect is that you will be good at using Git (I really do love git..) since you will be using it a lot. But one of the biggest earnings of taking this challenge is that you will be really good at navigating in the Linux kernel source code, which is necessary when doing kernel development. The concepts in the kernel stands, but the implementation vary with versions and so will the internal API, so you have to lookup code all the time. Fast navigation will facilitate for sure.

Personally I use VIM as editor with plugins for cscope/ctags/global for fast navigation and it works like a charm, really.
A tips is to use ”git grep” for searching within the project, it is much faster than a recursive ”grep” since it only searching in files which is part of the repository (not searching auto-generated crap).

I hope you find this challenge interesting and you give it a chance. The time is well spent if you ask me, and it will take time. The challenge will require a lot of studying and reading source code, but let it take the time it needs, this is no race.

See ya on the lkml!

PS: Here is just a quote from a random mail in my inbox… :-)

Very nice job, you are now done!
If you were curious, you are the 102nd person to complete the series of
tasks, with over 11000 people currently attempting it.

ldd without ldd

Av Marcus Folkesson den 14 Juni 2015

I sometimes meet colleges at work who gets frustrated when they try to print the shared libraries dependencies for an ELF or library, and the ldd command is simply stripped out from target. (I do often strip targets :-) )

As if that would be a big problem.

The ldd command is not a binary executable, but a script that simple calls the runtime dynamic linker with a few environment variables set, and you may do the same!
The essential environment variable in this case is LD_TRACE_LOADED_OBJECTS, that should be set to something != 0.

In short, you may do:
LD_TRACE_LOADED_OBJECTS=1 /lib/ ./my_application.

Even the –list option may be used, but does not work on all targets.

/lib/ --list ./my_application.

Example outputs with ldd:

[11:16:58]marcus@tuxie:/tmp/a$ ldd ./main =>  (0x00007fffc8bff000) => /lib/x86_64-linux-gnu/ (0x00007f564e254000) /lib64/ (0x00007f564e647000)

Example output without ldd:

[11:17:23]marcus@tuxie:/tmp/a$ LD_TRACE_LOADED_OBJECTS=1 /lib64/ ./main =>  (0x00007fffeecbc000) => /lib/x86_64-linux-gnu/ (0x00007fcce7700000) /lib64/ (0x00007fcce7af3000)

So, do not get frustrated, be happy.

Apple och öppenhet

Av Mats Sjövall den 8 Juni 2015

Haha, jag måste bara kommentera att Apple idag utannonserade att dom skulle OpenSource:a Swift. Lite skämtsamt förutspådde jag detta för ett år sen i en tidigare bloggpost och man kan faktiskt se en trend att de stora företagen slåss om oss utvecklare genom att göra oss delaktiga i deras produkter på ett sätt vi inte sett förrut. Jag tycker det är en mycket positiv trend och detta fick mig lite mer nyfiken att kolla närmare på Apple:s utvecklingsplattform.

What is your workflow strategy?

Av Johan Deimert den 31 Maj 2015

Have you ever thought of that? And additionally, how well does your working environment support your preferred workflow?

Me, for starters, have always been a great fan of the Visual Studio Development Environment. Primarily because this was the first encounter with an IDE that was setup for a large project, this was VS 6.0 and I worked as a contractor at Sony Ericsson Mobile Communications. I learned a lot about VS and how it could work with some carefully selected or written plugins.

And what I have come to realize is that what I really like about VS is the debugger and its environment. In my opinion hands down still the best debugger.

Fast forward and a lot of the later projects that I have been running has involved stuff that are not targeted for the Windows environment, i.e. setting up VS to work correctly is harder and sometimes impossible. So if I cannot have the functions as I want in an IDE how do you do it?
Well, some sort of text editor, cross compiler environment with Makefiles, a serial port and hopefully some gdb connection to be able to some decent debugging.

What this means for me and my workflow is when I start up the computer and log in I running my ”super-special-delux-script” that opens a number of terminals, some sort of light weight editor e.g. Geany and a web browser. All spread out over the virtual desktops. And usually there are some additional terminals opened along the way as well as some more editors, some remote desktops and puuhhh.

Some of you might say that Eclipse is the answer, I say meh… Every time I try Eclipse for a new project I get mad and I know why. The first encounter that I had with Eclipse and setting up a cross compiler environment was really bad and the person that was there to help me even worse. And that is still in my head many years later.

My new strategy is however to try to do things based on the terminal. So my new workflow strategy for development in Linux environment is:
- the i3 tiling window manager
- VIM, setting it up with the help found on Ben McCormick – Learning VIM
- new file structure on disk
- really trying to learn and understand gdb and gdbserver

It is coming along very nicely :-)
The best improvement I have noticed at this point is the i3 tiling windows manager, I think this has done some wonders for my workflow. And as a step to be more fluent in VIM I have changed the key layout to better map VIM keys. And taking it one step further installed the Vimium App in Chromium (installation link). So there is no excuse to not just the keyboard instead of the mouse. This is also a workflow strategy, training the muscle memory.

I still do use Eclipse for some projects, but if its sole purpose is the be a code editor I try to run VIM and get used to the interface.

Running for a trial period as of now I use the Jetbrains CLion, a code IDE that uses CMake to build the project. And I hope to use that IDE some more as well.

Tip of the day

Running i3, VIM and Vimium and other keyboardcontrolled applications the ESC key is to far off and I will have some pain in my left pinky at the end of a code intensive day. So I have, like many, many, many others before me, remapped the useless CAPS LOCK key to act as ESC.

# vim .i3/config

Add the following line some where in the end:

exec –no-startup-id ”setxkbmap -option caps:escape”

And Save

To next time, have a good one!

Debugging Segmentation faults

Av Marcus Folkesson den 31 Maj 2015

I’ve heard about people *hmph* that got segmentation faults with their code. For those people, there are several options to figure out what really happens. This post will discuss some techniques to tracking the fault down.
The code we will use do a simple access-violation when trying to write to address 0×0.


#include <stdlib.h>
void b()
char * n = NULL;
*n = 33;
void a()
int main(int argc, const char *argv[])
return 0;

Debugging with GDB

Of course, the application can be started with a debugger or the debugger may be attached to an already running process.
Compile the application with debug information and just start it:
gcc ./segfault.c -g -o segfault
gdb -tui ./segfault
gdb -tui ./segfault <PID>
to attach to a process.

However, as this is the standard way during the development cycle it will not be covered here.

Debugging with GDB and coredump

Linux may generate a coredump containing debug information about the state the application had when it crashed. It makes it possible to do backtraces, print variables, stacks and so on. The default size of the coredump is zero, so no dump will be generated by default. The value may be changed with the ulimit command:
ulimit -c unlimited
is to set the core-size. See ulimit -a for more details.

To disable the coredump generation, just set it back to 0.

Now if we start the application we will see that a coredump will be generated.
gcc ./segfault.c -g -o segfault
ulimit -c unlimited
Segmentation fault (core dumped)

Load the debug-information with GDB:
gdb -tui ./segfault ./core


This is really useful, but with large application the dump may grow really large and in an embedded Linux system, memory may be a limited resource. And yes, it is possible to set the coredump size to other than unlimited, but this will only result in a useless coredump when you really need it.


LD_PRELOAD is an environment variable that tells the dynamic loader (usually /lib/ to load a list of libraries before all other. This has the effect that when the loader tries to find symbols, it will look in the preloaded library first. If it find the symbol, it will stop there and look for the next symbol. This will result in that symbols may be overridden by the one in the preloaded library. This is useful in several cases, for example, the malloc function may be overridden to log all memory allocations. sets the signal handle for segfaults (SEGFAULT_SIGNALS may be set to handle more signals) and outputs a register dump, backtrace and a memory map if the signal got catched.
Start the application with ./segfault.

[09:20:30]marcus@tuxie:/tmp/a$ ./segfault
*** Segmentation fault
Register dump:
RAX: 0000000000000000   RBX: 0000000000000000   RCX: 0000000000000000
RDX: 00007ffc9fe25a08   RSI: 00007ffc9fe259f8   RDI: 0000000000000001
RBP: 00007ffc9fe258e0   R8 : 00007f4766767e80   R9 : 00007f4766982560
R10: 00007ffc9fe257a0   R11: 00007f47663c8dd0   R12: 0000000000400400
R13: 00007ffc9fe259f0   R14: 0000000000000000   R15: 0000000000000000
RSP: 00007ffc9fe258e0
RIP: 00000000004004fd   EFLAGS: 00010206
CS: 0033   FS: 0000   GS: 0000
Trap: 0000000e   Error: 00000006   OldMask: 00000000   CR2: 00000000
FPUCW: 0000037f   FPUSW: 00000000   TAG: 00000000
RIP: 00000000   RDP: 00000000
ST(0) 0000 0000000000000000   ST(1) 0000 0000000000000000
ST(2) 0000 0000000000000000   ST(3) 0000 0000000000000000
ST(4) 0000 0000000000000000   ST(5) 0000 0000000000000000
ST(6) 0000 0000000000000000   ST(7) 0000 0000000000000000
mxcsr: 1f80
XMM0:  0000000000000000000000000000ff00 XMM1:  0000000000000000000000000000ff00
XMM2:  0000000000000000000000000000ff00 XMM3:  0000000000000000000000000000ff00
XMM4:  0000000000000000000000000000ff00 XMM5:  0000000000000000000000000000ff00
XMM6:  0000000000000000000000000000ff00 XMM7:  0000000000000000000000000000ff00
XMM8:  0000000000000000000000000000ff00 XMM9:  0000000000000000000000000000ff00
XMM10: 0000000000000000000000000000ff00 XMM11: 0000000000000000000000000000ff00
XMM12: 0000000000000000000000000000ff00 XMM13: 0000000000000000000000000000ff00
XMM14: 0000000000000000000000000000ff00 XMM15: 0000000000000000000000000000ff00
Memory map:
00400000-00401000 r-xp 00000000 08:01 10747984                           /tmp/a/segfault
00600000-00601000 r–p 00000000 08:01 10747984                           /tmp/a/segfault
00601000-00602000 rw-p 00001000 08:01 10747984                           /tmp/a/segfault
00f65000-00f86000 rw-p 00000000 00:00 0                                  [heap]
7f4766191000-7f47661a7000 r-xp 00000000 08:01 12324878                   /lib/x86_64-linux-gnu/
7f47661a7000-7f47663a6000 —p 00016000 08:01 12324878                   /lib/x86_64-linux-gnu/
7f47663a6000-7f47663a7000 rw-p 00015000 08:01 12324878                   /lib/x86_64-linux-gnu/
7f47663a7000-7f4766562000 r-xp 00000000 08:01 12327383                   /lib/x86_64-linux-gnu/
7f4766562000-7f4766762000 —p 001bb000 08:01 12327383                   /lib/x86_64-linux-gnu/
7f4766762000-7f4766766000 r–p 001bb000 08:01 12327383                   /lib/x86_64-linux-gnu/
7f4766766000-7f4766768000 rw-p 001bf000 08:01 12327383                   /lib/x86_64-linux-gnu/
7f4766768000-7f476676d000 rw-p 00000000 00:00 0
7f476676d000-7f4766771000 r-xp 00000000 08:01 12327386                   /lib/x86_64-linux-gnu/
7f4766771000-7f4766970000 —p 00004000 08:01 12327386                   /lib/x86_64-linux-gnu/
7f4766970000-7f4766971000 r–p 00003000 08:01 12327386                   /lib/x86_64-linux-gnu/
7f4766971000-7f4766972000 rw-p 00004000 08:01 12327386                   /lib/x86_64-linux-gnu/
7f4766972000-7f4766995000 r-xp 00000000 08:01 12327380                   /lib/x86_64-linux-gnu/
7f4766b63000-7f4766b66000 rw-p 00000000 00:00 0
7f4766b92000-7f4766b94000 rw-p 00000000 00:00 0
7f4766b94000-7f4766b95000 r–p 00022000 08:01 12327380                   /lib/x86_64-linux-gnu/
7f4766b95000-7f4766b96000 rw-p 00023000 08:01 12327380                   /lib/x86_64-linux-gnu/
7f4766b96000-7f4766b97000 rw-p 00000000 00:00 0
7ffc9fe05000-7ffc9fe27000 rw-p 00000000 00:00 0                          [stack]
7ffc9fe47000-7ffc9fe49000 r–p 00000000 00:00 0                          [vvar]
7ffc9fe49000-7ffc9fe4b000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Segmentation fault (core dumped)

The most valuable information here is the backtrace. The backtrace gives us the address of where the segmentation fault occurred. addr2line is a tool to translate the address to a more useful line in the source code.
[10:24:48]marcus@tuxie:/tmp/a$ addr2line  -a 0x4004fd -e ./segfault

Debug without debug symbols

Run the application with debug symbol is not feasible  in all situation. The binary may be delivered to customer or the segmentation fault maybe only occur when compiled for release (those hard tracking bugs is a big part of my reality…). is still usable to get the information about where the code faults, but we have to disassemble the binary to see which instruction that caused it to happened.

First, disassemble the binary:
[10:47:05]marcus@tuxie:/tmp/a$ objdump ./segfault -DC > segfault.objdump

Then grep for the address in the disassemble:
[10:46:52]marcus@tuxie:/tmp/a$ grep -A10 -B10 4004fd ./segfault.objdump

4004e0:       e9 7b ff ff ff          jmpq   400460 <register_tm_clones>
4004e5:       0f 1f 00                nopl   (%rax)
4004e8:       e9 73 ff ff ff          jmpq   400460 <register_tm_clones>

00000000004004ed <b>:
4004ed:       55                      push   %rbp
4004ee:       48 89 e5                mov    %rsp,%rbp
4004f1:       48 c7 45 f8 00 00 00    movq   $0×0,-0×8(%rbp)
4004f8:       00
4004f9:       48 8b 45 f8             mov    -0×8(%rbp),%rax
4004fd:       c6 00 21                movb   $0×21,(%rax)
400500:       5d                      pop    %rbp
400501:       c3                      retq
0000000000400502 <a>:
400502:       55                      push   %rbp
400503:       48 89 e5                mov    %rsp,%rbp
400506:       b8 00 00 00 00          mov    $0×0,%eax
40050b:       e8 dd ff ff ff          callq  4004ed <b>
400510:       5d                      pop    %rbp
400511:       c3                      retq

Here we can see that the error occurred in the b function.
In this case it is quite simple to correlate the assembler instruction with the C line.
movb   $0×21,(%rax)
Here we move the value 0×21 (33 in decimal) to the variable that %rax is pointing to. It’s with the right high probability this line:
*n = 33;


Debugging is never a easy task but it helps to know your tools. A typical Linux environment already has these tools, so just start use them!