I have the following kernels installed:
- linux-zen (Zen)
- linux-rt (RealTime)
- linux-hardened (Security Hardened)
- linux-lts (Long Term Support)
- linux-tr-lts (Realtime LTS)
When I boot up, I try the different kernels from time to time just to see if anything interesting happens. It never does.
My question: How do I actually physically notice the difference between these kernels? If I use RT, does Firefox spawn quicker (in my testing, no, not really)?
What are some use cases when I can really see the difference in these kernels?
Difference goes from negligible to imperceptible. They were only really necessary for back in the day specific use cases like digital audio latency.
My question: How do I actually physically notice the difference between these kernels?
Generally, you don’t. You can look for some benchmark to try and find a difference between them, but if you don’t notice a difference in your day to day tasks, then it’s all the same. In my experience you should pick a kernel based on your desired experience. For my needs this is how the kernels differ:
- Generic kernel: a sane default for most regular users
- LTS: only makes sense if you’re worried about regressions in the generic kernel causing issues, and only viable if you can afford to stay behind on hardware driver updates, ie you use old hardware and/or optimal performance is not required
- Zen: sometimes better for gaming, but often indistinguishable from the generic kernel
- Realtime: rarely what you want, it sounds “faster” but it’s basically optimized for very specific use cases and if you’re not among them you’ll see the same or worse performance
I’m trying to tinker with my system and replace a perfectly good and well optimized default kernel for some kernel made for specific niche use cases and I don’t see any performance increase. Why would it be?
Yes, surprisingly the default kernel is optimized well rather than just being a badly written placeholder that users should manually replace for their system to become usable.
It’s 2025 and stuff is designed to just work out of the box.
Realtime is not about being fast, it’s about time guarantees. It helps with or is required for workloads that require realtime, which I think includes audio production, but might also be helpful for things like controllers etc. where you need to make sure incoming data is processed in a guaranteed time or else fail. Browsing the web isn’t part of these, so an RT kernel will most likely be a hindrance.
Yes, aircraft for example. If the pilot says “gear down”, the gear must go down in short order. You can’t say “well I think I’ll check the airspeed sensor a few times first”.
That’s pretty cool. Seems like a big undertaking tho
But it’s used in PES (Passenger Entertainment Systems) at least.
Ok this discussion reminds me of a gripe: Is there a Linux distribution or kernel that prioritizes the UI over everything else, including an OOM situation?
I’ve never had (modern) Windows kernel panic on me, or completely slow to the point I can’t get Ctrl+Alt+Del registered.
Let me know if I’m just using Linux stupidly though…
I’ve definitely had Windows hard lock before and stop responding to the keyboard, from Win95 all the way to Win10. I have no experience with Win11 so I can’t speak for that, but all others have situations where it can happen.
In fact, Windows is bad enough that the disk usage being high can cause the system to stop responding until it’s done and drops back down.
You can jam the Windows UI by spawning loads of processes with equivalent or higher priority to
explorer.exe, which runs the desktop as they’ll compete for CPU time. The same will happen if you do the equivalent under Linux. However if you have one process that does lots of small allocations, under Windows, once the memory and page file are exhausted, eventually an allocation will fail, and if the application’s not set up to handle that, it’ll die and you’ll have free memory again. Doing the same under every desktop Linux distro I’ve tried (which have mostly been Ubuntu-based, so others may handle it better) will just freeze the whole machine. I don’t know the details, but I’d guess it’s that the process gets suspended until its request can be fulfilled, so as long as there’s memory, it gets it eventually, but it never gets told to stop or murdered, so there’s no memory for things like the desktop environment to use.
There’s a systemd OOM service that can be setup. I use CachyOS and they have it as a checkbox to turn it on or off.
Going to look into this, thanks!
I thought that’s what Zen was for
I think banking is the old standard example of real time kernel needs. Money goes in, numbers go up, no time to explain the tide.
I don’t know if you’re being serious, but I can confirm from my time at as a developer at a banking software company, we didn’t use a hard RT OS even for like Mosler or Hitachi high speed check sorters. Just fast C++ code. (On Windows XP still, when I left in 2016)
(Work load is basically: batch of checks is loaded into an input hopper, along with check sized pieces of paper which are headers and footers, machine rapidly scans MICR lines and they go flying towards output pockets, and our code has something like 20 ms to receive the MICR data and pass back a sorting decision.)
Check or cheque?
My mind was on Unix systems with real time kernels from a time when Windows were but openings in the wall or roof of buildings or vehicles, fitted with glass in a frame to admit light or air and allow people to see out. And later Linux.
Zen and “mainline” (default/vanilla) are generally fine for “desktop use” and gaming. Zen is basically the mainline kernel with some tweaks. They are mostly concerned with latency, reducing the maximum time a process can spend blocking the processor - among other things.
This can lead to less input lag or a “smoother” desktop experience, but overall performance is as good as mainline at most. Slightly worse in some scenarios.
Hardned is a tradeoff afaik. You will stay behind mainline a bit, but get extra hardening. This can also impact performance, but rarely does in a meaningful way. If you don’t have any specific reason to use it, e.g. you carry it around on a laptop with sensitive data, I would look at other ways to harden my system first (firewall, encryption, access control, anti-virus, sandboxing, VPN…).
Pretty much the same goes for LTS, but with the focus more on stability than security.
RT is only for special applications.
From what I recall the completely fair scheduler (CFS) used by default on most Linux systems has a lower average latency than the RT kernel. The RT kernel just gives you more consistency, hence the CFS having lower latency “on average”
So honestly for opening Firefox it’ll probably depends more on your SSD data rate, but in theory it’ll open faster on a “regular” distribution most of the time.
Real time is good for things like audio processing where having better guarantees that a process will get its share of the CPU is a benefit.
Here is a nice video that gives you an easy to grasp intuition about durations of different operations and access of components of a computer (Cache vs RAM vs SSD vs HDD etc.)
I find it illustrates well why a fester drive or even faster RAM (unless there is a different bottleneck) would give you a more noticable performance uplift than a different Kernel.
Have you given the CachyOS kernel a try? It’s got some of the Clear Linux patches and some other custom patches, and it might have slightly better performance than the others you’ve listed here
Although expect to only really see any noticeable improvements in games or benchmarks and the like
I don’t think there will be a noticeable difference. Real Time kernel is intended for things like microcontrollers I believe.
Realtime is important on fully fledged workstations where timing is very important. Which is the case for a lot of professional audio workloads. Linux is now another option for people in that space.
Not sure Linux can run on microcontrollers. Those tend to not be so powerful and run simple OSs if they have any OS at all. Though this might help the embedded world a bit increasing the number of things you can do with things that have full system on chips (like the Raspberry pi).
So it’s more for dealing with hardware interruptions quickly, and likely will not help with gaming?
