The Synthesist

Starting-the-Kernel-Journey—My-LKMP-Experience

2025-12-08T00:00:00Z

Bound by the shackles of Knowledge and fear,
Linux Kernel had me rolling my tears.
Liberation I see was nowhere near,
My love for the kernel seemed to disappear.
Then came the LKMP, that ignited the spark,
Melting those shackles to free me at last.
Now I am free and over my fears,
Patching the kernel without any tears.
GK

Before I write about my experience, I would like to thank my mentors Shuah Khan, David Hunter and Khalid Aziz for providing me this opportunity to become a part of the LKMP - 2025 Fall Program and this wonderful kernel community.

What is LKMP?

LKMP stands for Linux Kernel Mentorship Program. Its actually a remote mentorship program that is conducted by Linux Foundation along with some of the best Kernel Engineers and Maintainers from the community to train the next generation of Linux Kernel developers.

The core objectives of the program are:

Provide remote learning oppurtunity to the aspiring Linux Kernel developers.
Infuse new talent into the Linux Kernel community.
Make the kernel more secure and sustainable.

There are a couple of variants of LKMP program but the one that is currently running is the Bug Fixing variant (Unpaid). You can find more information about the program here : lkmp

Acceptance Into The Program

Once you have created an account at LFX portal and enrolled in the program, you will be given a set of tasks, that you will have to complete within the stipulated deadline (around 3 weeks). These tasks might include writing kernel modules, analyzing syzbot bugs and sending kernel patches.
Along with that you will be also required to complete LFD103 Course
Once you have submitted all the tasks and completed the course, wait until the deadline :)

Please take the tasks pretty seriously. This time only 74 out 486 applicants were selected i.e around 15%. So make sure to give it your best.

Mentorship Info

Communication

For communicating with mentors and peers, we use discord
Patches and their reviews happen over the email.

Before getting selected, I began using neomutt for working with emails. I didnt like the existing themes in neomutt, so decided to submit a theme based on the ayu-dark color palette. That got merged. Therefore if you like ayu-dark theme, make sure to try it using the steps mentioned here.

Its just amazing that for a project Like linux kernel, email acts as the primary way of communication. Initially i was skeptical about how all of this works, but once you start using it, you see the real power of emails. Everything from sending patches, reviewing patches, applying patches becomes so much easier using email. Emails are just extremely flexible.

Meetings

There is a 1hr meeting scheduled everyweek (office hours), where you can clarify your doubts, get your patches reviewed. In the first few office hours, Shuah will also take some tutorial sessions on tools like csope/ctags, cregit, kselftests, syzbot, backporting etc.
Please make sure to attend those tutorial sessions. They will make your life easier and help you understand different aspects of kernel development including setting up a workflow, debugging, testing, finding bugs, backporting etc.
These office hours also allow you to understand how your peers are tackling problems. I was highly inspired by some of my peers during this process and this helped me fix my own problems as well.
David Hunter also conducted an additional session on Bug hunting in the kernel, which I found really effective. This was a hands on session where we discovered new bugs together. Thanks a lot David for this session.

Note Its just amazing, how much patience our mentors had when we got stuck on issues. Shuah and David both went above and beyond trying to help us fix our problems. They did not solve our issues directly, but they made sure to understand the problem and guide us in the right path. Again I cannot emphasize enough, how much that is helpful for mentees. Thanks a lot for helping us, inspiring us and giving us the confidence to submit our patches.

Community

The linux kernel community is beginner friendly in nature. Since we communicate through emails, there are mailing lists for each subsystem in the kernel. You can find a list of these mailing lists at lore.kernel.org

lore is an extremely useful web archive as you can find every patch in there. You can also see the threads associated with those patches. Along with that lore also provides you ways to search for patches that touch perticular file or are specific to perticular author, which can be extremely helpful.

Learning to interact with the community is a key part of kernel development. In order to improve your communication skills, you can go through the lore threads of the subsystem that you are interested in.

In addition, I also took LFC114 Course offered by linux foundation to help me improve in this area.

My Patches

Although I do have some experience working with the kernel as a part of a research project, I still consider myself as a novice in actual kernel development.

Therefore I decided to start slow and focus on submitting three kinds of patches in the kernel,

Fixing documentation build faliures
Fixing static analysis bugs reported by the compiler and internal kernel tools
Fixing syzbot bugs
git log --author="Gopi Krishna Menon" --since=“2025-08-01” --pretty --format=oneline a5160af78be7fcf3ade6caab0a14e349560c96d7 usb: raw-gadget: cap raw_io transfer length to KMALLOC_MAX_SIZE 89194773f5738a0617240ac92b190fbd553e58bc drm/msm: Add NULL check in vm_op_enqueue() 9efb297c520f392ab04bc45544a03770c98c3798 hwmon: (gpd-fan) Fix compilation error in non-ACPI builds 02227b97a8d3ae01bcf12e5e1b5cb6cca9439875 selftests: tty: add tty_tiocsti_test to .gitignore 77cd9210271556aff955551b3e2ef0ae8a2691bd docs: trusted-encrypted: fix htmldocs build error 96b546c241b11a97ba1247580208c554458e7866 Documentation/rtla: rename common_xxx.rst files to common_xxx.txt 7e8f305a081e22ce81aab7f7b9ce01437cbd38b3 docs/zh_CN: Fix malformed table 989ed3cad2fdb7921bf0f9f0f730e1bb065946c2 docs/zh_TW: Fix malformed table 03faea8466713f04a522c52c386124755be960bc selftests/net: add tcp_port_share to .gitignore d496b6f42eb0455caf5d8cb30cf1f01b7fc2a747 mtd: cfi: use struct_size() helper for cfiq allocation cb0c5a60a6f7dd7f35fa5b46fd04dfb66b37fc1e docs: perf: Fujitsu: Fix htmldocs build warnings and errors 5fdb877b4916a1eb2b2c85018c2311855893405b selftests/futex: Fix typos and grammar in futex_priv_hash 05f297c3e39f62f579458a59ab1b1c8bf3fc1c51 KVM: selftests: fix minor typo in cpumodel_subfuncs 70d476b63a1494337f2b9fec1e9b1cab2e6116d3 Documentation/rv: Fix minor typo in monitor_synthesis page 0e6bb6888791656c3973f26da3288323524573c0 docs: folio_queue: Fix minor typo in folio_queue page

There were a number of other dynamic analysis bugs that I attempted as well.

Some of them I almost solved, but the kernel developers sent patches before me 🙂

Things to remember as a kernel developer

Test all your patches. Shuah mentioned this in almost every meeting multiple times. Test your changes extensively and use tools like kselftest to ensure that nothing breaks. On top of this, mention your testing methodology in the email itself when you send the patch. Untested code can result in regressions which can be a headache for everyone including maintainers, authors and the user themselves.
If you dont have the hardware to test the patch, avoid making that patch. Patches can be marked as RFT (Request for Testing), but still its advised not to work on those kinds of patches.
Avoid top posting in emails and instead use interleaved posting or bottom posting when replying. (Posting Style)
Initially, we can work on different subsystems for understanding the workflow but after sometime, you should choose a subsystem of your interest and try to gravitate towards that. This will help you make bigger contributions to the kernel.
Write the commit messages in Imperative Tone
If your patch is not accepted yet, wait for a few weeks. Maintainers are also humans and therefore sometimes things might take time to get reviewed. Look out for the merge window as well. If you dont get a reply for say 2-3 weeks, gently ping the reviewers asking if they could look at the patch. (Just saying ping is a bad idea).
Read the documentation. Usually every subsystem has a slightly different way of accepting patches (they might require specific tags, specific tests to run etc). In that case check the mailing list for that perticular subsystem and see how others have done that.

How to find Bugs in the kernel?

Looking for Documentation Bugs? Run make html-docs
Looking for Static Analysis Bugs?
- Use Sparse, Smatch, Coccinelle
Looking for Dynamic Analysis Bugs?
- syzbot
- Additional tools can be found here

There is one more way. randconfig (For static analysis and build related bugs)

Here is a small script that will run randconfig n number of times and capture the problematic kernel builds.

In simple terms

First it generates a random config for the architecture
Then it builds the kernel using this config
If the build breaks or config has some issues, it logs down the config and the build output so that a human like me can review them and use them for fixing those bugs.

Note : Sometimes this does result in false positives. So be careful about the errors that you are investigating.

randconfig builds should atleast compile. If they fail to do so, that means there is some undeclared dependency and could be a problem in some configuration for some users.

Infact kernel's CI service also does something similar I believe.

#!/bin/bash
# ================================================
# Sequential randconfig builds for linux-next(x86-64)
# ================================================

BUILD_COUNT=10
ARCH="x86_64"
BASE_DIR=/linux_work
LINUX_NEXT_DIR=$BASE_DIR/linux
DATE=`date +"%d-%m-%y"`
LOG_DIR=$BASE_DIR/randconfig-logs/$DATE
CONFIG_FILE=.config
REMOTE=origin
BRANCH=master
NPROC=$(( $(nproc) / 2 ))

# Create log dir if not available
mkdir -p $LOG_DIR
echo "Starting $BUILD_COUNT sequential randconfig builds for $ARCH..."
echo "Logs will be stored in $LOG_DIR"

# Get the latest commits
cd $LINUX_NEXT_DIR
git fetch $REMOTE 
git reset --hard $REMOTE/$BRANCH
COMMIT_INFO=$(git log -1 --oneline)

# All sequential builds
for i in $(seq 1 $BUILD_COUNT); do
    LOG_FILE=build_log.log
    CONFIG_GEN_LOG_FILE=config_gen_log.log
    CONFIG_GEN_LOG_DIR=$LOG_DIR/config-gen-log

    echo "============================="
    echo "Build $i / $BUILD_COUNT"
    echo "============================="

    # Clean old build artifacts
    make mrproper > /dev/null

    # Remove previous logs.
    rm -rf $LOG_FILE
    rm -rf $CONFIG_GEN_LOG_FILE


    # Add the commit info in config_gen_log
    echo $COMMIT_INFO > $CONFIG_GEN_LOG_FILE
    # Generate random config
    make ARCH=$ARCH randconfig >> $CONFIG_GEN_LOG_FILE 2>&1 


    # For unique config name
    CONFIG_HASH=$(sha1sum .config | cut -c1-8)

    # Do we have warning or error when generating config? Log seperately
    if grep -iq "error:" "$CONFIG_GEN_LOG_FILE" || grep -iq "warning:" "$CONFIG_GEN_LOG_FILE"; then
      mkdir -p $CONFIG_GEN_LOG_DIR
      # Copy over the config into config-gen-log folder as config-hash
      cp $CONFIG_FILE $CONFIG_GEN_LOG_DIR/config-$CONFIG_HASH
    fi

    # Add the commit info in build log
    echo $COMMIT_INFO > $LOG_FILE
    # Build the kernel
    make -j$NPROC >> $LOG_FILE 2>&1

    # Check for errors or warnings
    if grep -iq "error:" "$LOG_FILE" || grep -iq "warning:" "$LOG_FILE"; then
      # We store the config file and build log
      ERROR_CONFIG_FILENAME=error_config
      ERROR_BUILD_LOG_FILENAME=error_build_log.log

      # We store the above files in a seperate dir
      ERROR_LOG_DIR=$LOG_DIR/config-$CONFIG_HASH
      mkdir -p $ERROR_LOG_DIR

      # Store the config and build log in here
      mv $CONFIG_FILE $ERROR_LOG_DIR/$ERROR_CONFIG_FILENAME
      mv $LOG_FILE $ERROR_LOG_DIR/$ERROR_BUILD_LOG_FILENAME
      echo "❌  Build $i failed. See $ERROR_LOG_DIR"
    else
      echo "✅  Build $i completed successfully."
    fi
done

echo "All $BUILD_COUNT builds finished."
echo "Logs are in $LOG_DIR"

The output will look something like this where both the config and the output can be used to reproduce the error.

Here error_config represents the kernel config that generates this issue
error_build_log represents the build log which shows the type of error encountered

Bonus Tip : If you wish to run this for different architectures, you can contact the guys over at GCC Compile Farm Project and request access to their machines for doing testing.

Future

I have found my interest in the USB and MM subsystem and wish to make substantial contributions in these areas in the Linux Kernel. Also I would like to become a maintainer at some point in the linux kernel.

If you are interested to get involved with the kernel community and aspire to be a linux kernel developer, I would highly recommend applying for the Linux Kernel Mentorship Program.

Happy Coding !!

It's not that we don't dare do things because they are difficult; rather, they are difficult because we don't dare – Seneca. (From Richard Stallman's personal site)

Dissecting a Simple Linux Kernel Module

2025-07-30T19:00:00Z

Dissecting a Simple Linux Kernel Module

Hi, In this short article we will explore the internal structure of a simple hello world module in the linux kernel, breaking down its anatomy and understanding how it integrates with the kernel.

Environment

Kernel Version : 6.16.0
Build Toolchain : clang 18, LLVM 18
OS : Ubuntu 22.04 / Nixos (Any distro will work)

Basics

The Linux kernel is monolithic in nature. Monolithic kernels lack the extensibility and modularity, which can make the kernel bloated and difficult to manage. Linux kernel solves this problem with kernel modules.

A kernel module is an object file containing code that can be loaded at runtime and can be used to extend the functionality of the kernel. When a module is no longer needed, it can be automatically unloaded, thereby reducing the memory footprint. Modules give several advantages to the kernel, such as extensibility, modularity, low memory footprint, faster compilation, etc. Most of the device drivers in the kernel are implemented in the form of modules, which contributes significantly to the modularity and maintainability of the kernel.

Internally, the kernel consists of a loader that, when requested, can load and map this object file into its own address space. Once mapped, the module can call kernel symbols just like any other part of the kernel.

Having a modular kernel also allows companies to provide their drivers in the form of proprietary binary blobs. This way, the company can apply its licenses to the source code and is not obligated to release the source code. Whether it's a good thing or a bad thing is a different topic (in terms of philosophy, security, etc.), but I guess companies have their reason for doing this.

Source Code

Let us now examine the traditional hello_world module and understand what happens internally.

// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>

MODULE_DESCRIPTION("A simple hello world module");
MODULE_AUTHOR("Gopi Krishna Menon");
MODULE_LICENSE("GPL");

static int __init hello_mod_init(void)
{
        pr_info("%s: Hello World\n", KBUILD_MODNAME);
        return 0;
}

static void __exit hello_mod_exit(void)
{
        pr_info("%s: Unloading hello_mod\n", KBUILD_MODNAME);
}

module_init(hello_mod_init);
module_exit(hello_mod_exit);

The code illustrates the fundamental structure of a kernel module. Its key components are :

License Identifier (SPDX Identifier)
HeaderFile (module.h)
Module Metadata
Initialization and Cleanup Functions - Entry and Exit Points

License Identifier

The first line // SPDX-License-Identifier: GPL-2.0 indicates the license associated with the source file. Although linux kernel is primarily licensed under GPL, there are several drivers that have a different free license such as MIT, BSD etc. SPDX or Software Package Data Exchange is a freely available open standard that provides a clean and unambiguous way (one line) to indicate the license associated with the source file. If we didnt have the SPDX standard, the license for each of these source files would look something like this :

/*
 * Copyright (C) 2025 Meow
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License...
 */

which is harder to read and can be ambiguous sometimes.

Header File

#include <linux/module.h>

This header provides the necessary kernel API's and macros to create a kernel module. MODULE_DESCRIPTION, MODULE_AUTHOR and MODULE_LICENSE and several other macros are defined within this header file.

Module Metadata

The macros MODULE_DESCRIPTION, MODULE_AUTHOR and MODULE_LICENSE define metadata about the module. This metadata is then embedded into the resulting .ko file that can be read by tools like modinfo or the kernel module loader.

So lets have a look at what these macros distill down to ultimately.

Looking at the definition of MODULE_DESCRIPTION, MODULE_AUTHOR or MODULE_LICENSE we can see that all of these macros boil down to MODULE_INFO.

/* Generic info of form tag = "info" */
#define MODULE_INFO(tag, info) __MODULE_INFO(tag, tag, info)

Here we can observe from the comment that the module info is represented in the form of tag:value pairs.

#define __MODULE_INFO(tag, name, info)                    \
    static const char __UNIQUE_ID(name)[]                 \
        __used __section(".modinfo") __aligned(1)         \
        = __MODULE_INFO_PREFIX __stringify(tag) "=" info

Here UNIQUE_ID ensures that the variable names never clash. Basically it appends UNIQUE_ID prefix to the variable and a unique number as the suffix to the variable. This is necessary because, if we look at the previous code snippet we can see that we pass tag as the name parameter to the macro. Therefore if invoke the macro multiple times (like multiple authors), then it could result in variable name collision which can be avoided using this.

But the variable name is not important here. Whats important is that the data of this variable is added into .modinfo section and the data itself is tag=info. Now modinfo section is very important as it contains the module information.

In case of our hello world module, if we generate the LLVM IR for this module, we can see that this is generated.

To generate the IR for this kernel module what I did is

make V=1 LLVM=1 M=custom_modules/hello_world

From this I extracted the line containing clang invocation and copied that to a seperate bash script where i gave emit-llvm -S option to clang and specified it to generate hello_world.ll file.

@__UNIQUE_ID_description481 = internal constant [40 x i8] c"description=A simple hello world module\00", section ".modinfo", align 1, !dbg !0
@__UNIQUE_ID_author482 = internal constant [26 x i8] c"author=Gopi Krishna Menon\00", section ".modinfo", align 1, !dbg !7
@__UNIQUE_ID_author483 = internal constant [12 x i8] c"author=Meow\00", section ".modinfo", align 1, !dbg !14
@__UNIQUE_ID_license484 = internal constant [12 x i8] c"license=GPL\00", section ".modinfo", align 1, !dbg !19

As we can observe, the data is stored in the form of tag:value pairs. Also I added additional author to show you how UNIQUE_ID is helping here. Now if we look at the hexdump of .modinfo section via readelf, we can see that the data is stored in the same manner inside the section:

readelf -x .modinfo custom_modules/hello_world/hello_world.k

Hex dump of section '.modinfo':
  0x00000000 64657363 72697074 696f6e3d 41207369 description=A si
  0x00000010 6d706c65 2068656c 6c6f2077 6f726c64 mple hello world
  0x00000020 206d6f64 756c6500 61757468 6f723d47  module.author=G
  0x00000030 6f706920 4b726973 686e6120 4d656e6f opi Krishna Meno
  0x00000040 6e006c69 63656e73 653d4750 4c006e61 n.license=GPL.na
  0x00000050 6d653d68 656c6c6f 5f776f72 6c640064 me=hello_world.d
  0x00000060 6570656e 64733d00 73726376 65727369 epends=.srcversi
  0x00000070 6f6e3d32 46374643 31393532 44453044 on=2F7FC1952DE0D
  0x00000080 44453443 30454638 45370076 65726d61 DE4C0EF8E7.verma
  0x00000090 6769633d 362e3136 2e30636c 616e672b gic=6.16.0clang+
  0x000000a0 20534d50 20707265 656d7074 206d6f64  SMP preempt mod
  0x000000b0 5f756e6c 6f616420 6d6f6476 65727369 _unload modversi
  0x000000c0 6f6e7320 00726574 706f6c69 6e653d59 ons .retpoline=Y
  0x000000d0 00

Running modinfo, we can see the .modinfo data

modinfo custom_modules/hello_world/hello_world.ko
filename:       /linux_work/linux-clang/custom_modules/hello_world/hello_world.ko
description:    A simple hello world module
author:         Gopi Krishna Menon
license:        GPL
name:           hello_world
depends:
srcversion:     2F7FC1952DE0DDE4C0EF8E7
vermagic:       6.16.0clang+ SMP preempt mod_unload modversions
retpoline:      Y

Note : In the above code you can observe that there is vermagic, which is also present in the .modinfo section. Basically if you load a module and vermagic of module does not match whats there in the kernel, it wont load the module. =So modules built for one kernel is not compatible with other kernel.=

static int check_modinfo(struct module *mod, struct load_info *info, int flags)
{
    const char *modmagic = get_modinfo(info, "vermagic");
    int err;

    if (flags & MODULE_INIT_IGNORE_VERMAGIC)
        modmagic = NULL;

    /* This is allowed: modprobe --force will invalidate it. */
    if (!modmagic) {
        err = try_to_force_load(mod, "bad vermagic");
        if (err)
            return err;
    } else if (!same_magic(modmagic, vermagic, info->index.vers)) {
        pr_err("%s: version magic '%s' should be '%s'\n",
               info->name, modmagic, vermagic);
        return -ENOEXEC;
    }

    err = check_modinfo_livepatch(mod, info);
    if (err)
        return err;

    return 0;
}

Module License Enforcement

The MODULE_LICENCE macro is not just for display, it has runtime implications. Basically - If you declare MODULE_LICENCE(GPL), the module is treated as GPL compatible - If you declare MODULE_LICENCE(Proprietary) or leave blank, it is not GPL compatible (meaning you cant access kernel symbols which are marked GPL only using EXPORT_SYMBOL_GPL).

Whatever license we have supplied to this macro can impact what symbols can we access from the kernel. So basically if we have used PROPRIETARY licence, then we cannot call or access any symbols from the kernel which are marked GPL only (using EXPORT_SYMBOL).

Initialization and Cleanup Functions

The functions hello_mod_init and hello_mod_exit are initialization and cleanup routines for the module.

static int __init hello_mod_init(void)
{
    pr_info("%s: Hello World\n", KBUILD_MODNAME);
    return 0;
}

static void __exit hello_mod_exit(void)
{
    pr_info("%s: Unloading hello_mod\n", KBUILD_MODNAME);
}

You can think of these as analogous to setup and teardown functions. - hello_mod_init is called when the module is inserted into the kernel. It can be used for setting up resources required for this perticular module - hello_mod_exit is called when the module is removed from the kernel. It can be used for cleaning up any resources acquired during initialization.

Inside the init and exit function, I am simply printing "Hello World" and "Unloading hello_mod" to the kernel ring buffer. Return value of 0 indicates success as usual. If the initialization fails for some reason, you can also return the error code such as -ENOMEM, -EINVAL which the kernel will handle accordingly.

The __init and __exit are annotations which instruct the compiler to place these functions in special section .init.text and .exit.text. The kernel can take this as a hint and can remove the code in these sections when the initialisation or removal is complete. This can help freeing up memory resources.

Finally the macros module_init and module_exit are used to register these functions. module_init basically specifies the driver initialisation entry point. If the module is builtin (compiled into the kernel), then it will be invoked during do_initcalls(), otherwise it will be invoked by do_init_module during module loading

...
    freeinit->init_text = mod->mem[MOD_INIT_TEXT].base;
    freeinit->init_data = mod->mem[MOD_INIT_DATA].base;
    freeinit->init_rodata = mod->mem[MOD_INIT_RODATA].base;

    do_mod_ctors(mod);
    /* Start the module */
    if (mod->init != NULL)
        ret = do_one_initcall(mod->init); /* Here */
    if (ret < 0) {
        goto fail_free_freeinit;
    }
    if (ret > 0) {
        pr_warn("%s: '%s'->init suspiciously returned %d, it should "
            "follow 0/-E convention\n"
            "%s: loading module anyway...\n",
            __func__, mod->name, ret, __func__);
        dump_stack();
    }

    /* Now it's a first class citizen! */
    mod->state = MODULE_STATE_LIVE;
    blocking_notifier_call_chain(&module_notify_list,
                     MODULE_STATE_LIVE, mod);


...

For exit, if it is not built into the kernel, it will be invoked by delete_module syscall which is invoked by rmmod

...
    /* If it has an init func, it must have an exit func to unload */
    if (mod->init && !mod->exit) {
        forced = try_force_unload(flags);
        if (!forced) {
            /* This module can't be removed */
            ret = -EBUSY;
            goto out;
        }
    }

    ret = try_stop_module(mod, flags, &forced);
    if (ret != 0)
        goto out;

    mutex_unlock(&module_mutex);
    /* Final destruction now no one is using it. */
    if (mod->exit != NULL)
        mod->exit(); /* HERE */
    blocking_notifier_call_chain(&module_notify_list,
                     MODULE_STATE_GOING, mod);
    klp_module_going(mod);
    ftrace_release_mod(mod);
...

So yea, this completes the dissection of a simple kernel module, in future articles, I will talk about other macros such as module_param, module_softdep, module_weakdep etc.