🧩 The Synthesist

Starting the Kernel Journey - My LKMP Experience

10 min read

Bound by the shackles of Knowledge and fear,
Linux Kernel had me rolling my tears.

Liberation I see was nowhere near,
My love for the kernel seemed to disappear.

Then came the LKMP, that ignited the spark,
Melting those shackles to free me at last.

Now I am free and over my fears,
Patching the kernel without any tears.
- GK

Before I write about my experience, I would like to thank my mentors Shuah Khan, David Hunter and Khalid Aziz for providing me this opportunity to become a part of the LKMP - 2025 Fall Program and this wonderful kernel community.

What is LKMP?

LKMP stands for Linux Kernel Mentorship Program. Its actually a remote mentorship program that is conducted by Linux Foundation along with some of the best Kernel Engineers and Maintainers from the community to train the next generation of Linux Kernel developers.

The core objectives of the program are:

  • Provide remote learning oppurtunity to the aspiring Linux Kernel developers.
  • Infuse new talent into the Linux Kernel community.
  • Make the kernel more secure and sustainable.

There are a couple of variants of LKMP program but the one that is currently running is the Bug Fixing variant (Unpaid). You can find more information about the program here : lkmp

Acceptance Into The Program

  • Once you have created an account at LFX portal and enrolled in the program, you will be given a set of tasks, that you will have to complete within the stipulated deadline (around 3 weeks). These tasks might include writing kernel modules, analyzing syzbot bugs and sending kernel patches.

  • Along with that you will be also required to complete LFD103 Course

  • Once you have submitted all the tasks and completed the course, wait until the deadline :)

Note

Please take the tasks pretty seriously. This time only 74 out 486 applicants were selected i.e around 15%. So make sure to give it your best.

Mentorship Info

Communication

  • For communicating with mentors and peers, we use discord
  • Patches and their reviews happen over the email.

Before getting selected, I began using neomutt for working with emails. I didnt like the existing themes in neomutt, so decided to submit a theme based on the ayu-dark color palette. That got merged. Therefore if you like ayu-dark theme, make sure to try it using the steps mentioned here.

Its just amazing that for a project Like linux kernel, email acts as the primary way of communication. Initially i was skeptical about how all of this works, but once you start using it, you see the real power of emails. Everything from sending patches, reviewing patches, applying patches becomes so much easier using email. Emails are just extremely flexible.

Meetings

  • There is a 1hr meeting scheduled everyweek (office hours), where you can clarify your doubts, get your patches reviewed. In the first few office hours, Shuah will also take some tutorial sessions on tools like csope/ctags, cregit, kselftests, syzbot, backporting etc.

  • Please make sure to attend those tutorial sessions. They will make your life easier and help you understand different aspects of kernel development including setting up a workflow, debugging, testing, finding bugs, backporting etc.

  • These office hours also allow you to understand how your peers are tackling problems. I was highly inspired by some of my peers during this process and this helped me fix my own problems as well.

  • David Hunter also conducted an additional session on Bug hunting in the kernel, which I found really effective. This was a hands on session where we discovered new bugs together. Thanks a lot David for this session.

Note

Its just amazing, how much patience our mentors had when we got stuck on issues. Shuah and David both went above and beyond trying to help us fix our problems. They did not solve our issues directly, but they made sure to understand the problem and guide us in the right path. Again I cannot emphasize enough, how much that is helpful for mentees. Thanks a lot for helping us, inspiring us and giving us the confidence to submit our patches.

Community

The linux kernel community is beginner friendly in nature. Since we communicate through emails, there are mailing lists for each subsystem in the kernel. You can find a list of these mailing lists at lore.kernel.org

lore is an extremely useful web archive as you can find every patch in there. You can also see the threads associated with those patches. Along with that lore also provides you ways to search for patches that touch perticular file or are specific to perticular author, which can be extremely helpful.

Learning to interact with the community is a key part of kernel development. In order to improve your communication skills, you can go through the lore threads of the subsystem that you are interested in.

In addition, I also took LFC114 Course offered by linux foundation to help me improve in this area.

My Patches

Although I do have some experience working with the kernel as a part of a research project, I still consider myself as a novice in actual kernel development.

Therefore I decided to start slow and focus on submitting three kinds of patches in the kernel,

  • Fixing documentation build faliures
  • Fixing static analysis bugs reported by the compiler and internal kernel tools
  • Fixing syzbot bugs
git log --author="Gopi Krishna Menon" --since=“2025-08-01” --pretty --format=oneline
a5160af78be7fcf3ade6caab0a14e349560c96d7 usb: raw-gadget: cap raw_io transfer length to KMALLOC_MAX_SIZE
89194773f5738a0617240ac92b190fbd553e58bc drm/msm: Add NULL check in vm_op_enqueue()
9efb297c520f392ab04bc45544a03770c98c3798 hwmon: (gpd-fan) Fix compilation error in non-ACPI builds
02227b97a8d3ae01bcf12e5e1b5cb6cca9439875 selftests: tty: add tty_tiocsti_test to .gitignore
77cd9210271556aff955551b3e2ef0ae8a2691bd docs: trusted-encrypted: fix htmldocs build error
96b546c241b11a97ba1247580208c554458e7866 Documentation/rtla: rename common_xxx.rst files to common_xxx.txt
7e8f305a081e22ce81aab7f7b9ce01437cbd38b3 docs/zh_CN: Fix malformed table
989ed3cad2fdb7921bf0f9f0f730e1bb065946c2 docs/zh_TW: Fix malformed table
03faea8466713f04a522c52c386124755be960bc selftests/net: add tcp_port_share to .gitignore
d496b6f42eb0455caf5d8cb30cf1f01b7fc2a747 mtd: cfi: use struct_size() helper for cfiq allocation
cb0c5a60a6f7dd7f35fa5b46fd04dfb66b37fc1e docs: perf: Fujitsu: Fix htmldocs build warnings and errors
5fdb877b4916a1eb2b2c85018c2311855893405b selftests/futex: Fix typos and grammar in futex_priv_hash
05f297c3e39f62f579458a59ab1b1c8bf3fc1c51 KVM: selftests: fix minor typo in cpumodel_subfuncs
70d476b63a1494337f2b9fec1e9b1cab2e6116d3 Documentation/rv: Fix minor typo in monitor_synthesis page
0e6bb6888791656c3973f26da3288323524573c0 docs: folio_queue: Fix minor typo in folio_queue page

There were a number of other dynamic analysis bugs that I attempted as well.

Some of them I almost solved, but the kernel developers sent patches before me 🙂

Things to remember as a kernel developer

  • Test all your patches. Shuah mentioned this in almost every meeting multiple times. Test your changes extensively and use tools like kselftest to ensure that nothing breaks. On top of this, mention your testing methodology in the email itself when you send the patch. Untested code can result in regressions which can be a headache for everyone including maintainers, authors and the user themselves.

  • If you dont have the hardware to test the patch, avoid making that patch. Patches can be marked as RFT (Request for Testing), but still its advised not to work on those kinds of patches.

  • Avoid top posting in emails and instead use interleaved posting or bottom posting when replying. (Posting Style)

  • Initially, we can work on different subsystems for understanding the workflow but after sometime, you should choose a subsystem of your interest and try to gravitate towards that. This will help you make bigger contributions to the kernel.

  • Write the commit messages in Imperative Tone

  • If your patch is not accepted yet, wait for a few weeks. Maintainers are also humans and therefore sometimes things might take time to get reviewed. Look out for the merge window as well. If you dont get a reply for say 2-3 weeks, gently ping the reviewers asking if they could look at the patch. (Just saying ping is a bad idea).

  • Read the documentation. Usually every subsystem has a slightly different way of accepting patches (they might require specific tags, specific tests to run etc). In that case check the mailing list for that perticular subsystem and see how others have done that.

How to find Bugs in the kernel?

  • Looking for Documentation Bugs? Run make html-docs
  • Looking for Static Analysis Bugs?
    • Use Sparse, Smatch, Coccinelle
  • Looking for Dynamic Analysis Bugs?

There is one more way. randconfig (For static analysis and build related bugs)

Here is a small script that will run randconfig n number of times and capture the problematic kernel builds.

In simple terms

  • First it generates a random config for the architecture
  • Then it builds the kernel using this config
  • If the build breaks or config has some issues, it logs down the config and the build output so that a human like me can review them and use them for fixing those bugs.

Note : Sometimes this does result in false positives. So be careful about the errors that you are investigating.

randconfig builds should atleast compile. If they fail to do so, that means there is some undeclared dependency and could be a problem in some configuration for some users.

Infact kernel’s CI service also does something similar I believe.

#!/bin/bash
# ================================================
# Sequential randconfig builds for linux-next(x86-64)
# ================================================
 
BUILD_COUNT=10
ARCH="x86_64"
BASE_DIR=/linux_work
LINUX_NEXT_DIR=$BASE_DIR/linux
DATE=`date +"%d-%m-%y"`
LOG_DIR=$BASE_DIR/randconfig-logs/$DATE
CONFIG_FILE=.config
REMOTE=origin
BRANCH=master
NPROC=$(( $(nproc) / 2 ))
 
# Create log dir if not available
mkdir -p $LOG_DIR
echo "Starting $BUILD_COUNT sequential randconfig builds for $ARCH..."
echo "Logs will be stored in $LOG_DIR"
 
# Get the latest commits
cd $LINUX_NEXT_DIR
git fetch $REMOTE 
git reset --hard $REMOTE/$BRANCH
COMMIT_INFO=$(git log -1 --oneline)
 
# All sequential builds
for i in $(seq 1 $BUILD_COUNT); do
    LOG_FILE=build_log.log
    CONFIG_GEN_LOG_FILE=config_gen_log.log
    CONFIG_GEN_LOG_DIR=$LOG_DIR/config-gen-log
 
    echo "============================="
    echo "Build $i / $BUILD_COUNT"
    echo "============================="
 
    # Clean old build artifacts
    make mrproper > /dev/null
 
    # Remove previous logs.
    rm -rf $LOG_FILE
    rm -rf $CONFIG_GEN_LOG_FILE
 
 
    # Add the commit info in config_gen_log
    echo $COMMIT_INFO > $CONFIG_GEN_LOG_FILE
    # Generate random config
    make ARCH=$ARCH randconfig >> $CONFIG_GEN_LOG_FILE 2>&1 
 
    
    # For unique config name
    CONFIG_HASH=$(sha1sum .config | cut -c1-8)
 
    # Do we have warning or error when generating config? Log seperately
    if grep -iq "error:" "$CONFIG_GEN_LOG_FILE" || grep -iq "warning:" "$CONFIG_GEN_LOG_FILE"; then
      mkdir -p $CONFIG_GEN_LOG_DIR
      # Copy over the config into config-gen-log folder as config-hash
      cp $CONFIG_FILE $CONFIG_GEN_LOG_DIR/config-$CONFIG_HASH
    fi
 
    # Add the commit info in build log
    echo $COMMIT_INFO > $LOG_FILE
    # Build the kernel
    make -j$NPROC >> $LOG_FILE 2>&1
 
    # Check for errors or warnings
    if grep -iq "error:" "$LOG_FILE" || grep -iq "warning:" "$LOG_FILE"; then
      # We store the config file and build log
      ERROR_CONFIG_FILENAME=error_config
      ERROR_BUILD_LOG_FILENAME=error_build_log.log
 
      # We store the above files in a seperate dir
      ERROR_LOG_DIR=$LOG_DIR/config-$CONFIG_HASH
      mkdir -p $ERROR_LOG_DIR
      
      # Store the config and build log in here
      mv $CONFIG_FILE $ERROR_LOG_DIR/$ERROR_CONFIG_FILENAME
      mv $LOG_FILE $ERROR_LOG_DIR/$ERROR_BUILD_LOG_FILENAME
      echo "❌  Build $i failed. See $ERROR_LOG_DIR"
    else
      echo "âś…  Build $i completed successfully."
    fi
done
 
echo "All $BUILD_COUNT builds finished."
echo "Logs are in $LOG_DIR"

The output will look something like this where both the config and the output can be used to reproduce the error.

  • Here error_config represents the kernel config that generates this issue
  • error_build_log represents the build log which shows the type of error encountered

Bonus Tip : If you wish to run this for different architectures, you can contact the guys over at GCC Compile Farm Project and request access to their machines for doing testing.

Future

I have found my interest in the USB and MM subsystem and wish to make substantial contributions in these areas in the Linux Kernel. Also I would like to become a maintainer at some point in the linux kernel.

If you are interested to get involved with the kernel community and aspire to be a linux kernel developer, I would highly recommend applying for the Linux Kernel Mentorship Program.

Happy Coding !!

It’s not that we don’t dare do things because they are difficult; rather, they are difficult because we don’t dare — Seneca. (From Richard Stallman’s personal site)

Graph View

Backlinks