Operating systems · Fundamentals

Introduction to operating systems

Kernel, syscalls, and the hardware contract

Start here

If you have never studied OSes before, read the definition and first-time map below first. They introduce words (kernel, syscall, privilege) that every later section reuses. You do not need assembly or C yet, only patience with layered explanations.

What is an operating system? (formal picture)

An operating system (OS) is a layer of trusted software that sits between application programs and the physical machine. Its job is to multiplex limited hardware (one or a few CPUs, finite RAM, shared disks and devices) so that many programs appear to run at once, while enforcing rules: who may touch which memory, which files, and which devices.

The kernel is the privileged core of the OS: it runs in a CPU mode that can configure the memory management unit (MMU), handle interrupts, and schedule threads. User space is where normal applications run; they must request privileged work through a narrow, audited interface called system calls. That split-untrusted user code vs trusted kernel code: is the central design idea of this entire course.

Big picture for first-time learners

Programs do not “talk to the disk” directly. They ask the kernel to perform actions on their behalf. The kernel checks permissions, updates internal data structures, and may put your thread to sleep until a disk interrupt completes.

Program vs process

A program is a file on disk (instructions + data). A process is a running instance: it has its own address space, registers, open files, and scheduling state. The OS creates processes, pauses them, and destroys them.

Kernel vs shell

The kernel enforces hardware rules. The shell (bash, zsh, etc.) is just another program that uses syscalls to start others. Confusing shell built-ins with kernel behavior is a common early mistake.

Why virtual memory exists

Each process typically sees a private range of addresses. The MMU translates those virtual addresses to physical RAM under kernel control. That gives isolation: one buggy program cannot scribble another process’s memory by accident.

Interrupts and concurrency

CPUs execute instructions sequentially, but interrupts from timers and devices let the kernel regain control, update I/O state, and wake sleeping threads. Scheduling and device handling depend on this mechanism.

Dual-mode operation (theory)

Modern processors expose at least two privilege levels. Mainstream Unix-like OSes use a simple model: user mode (restricted) and kernel mode (fully privileged for that OS’s definition of “kernel”).

When your code executes a trap instruction for a syscall, the CPU raises privilege, jumps to a kernel entry point, and runs handler code on a per-thread kernel stack. When the handler finishes, privilege is lowered and execution resumes in user space. A page fault is another trap: it may be benign (e.g. demand paging) or fatal (SIGSEGV).

Mechanism vs policy: the kernel provides mechanisms (runqueues, page tables, syscall tables). Policies (which process runs next, how much memory a cgroup may use) are configured by the same kernel but are conceptually separate, interviews often reward stating which side you mean.

Mental model to keep

Think of the OS as a referee: it does not “do your math for you,” but it decides whose turn it is to use the CPU, where results may be stored in memory, and how I/O completes safely.

By the end of this page you should be able to

Explain why user programs cannot install new page tables directly.
Name at least four resources the OS multiplexes (CPU, RAM, disk, devices, …).
Trace a high-level path from a library call (e.g. `open`) to a syscall and back.
Contrast mechanism with policy using one scheduling example.

Course summary (narrative)

Operating systems sit between bare metal and applications. They create the illusion of private memory and infinite CPUs by time-slicing and address translation, persist data through hierarchical file systems, and protect the machine with privilege levels and access checks. Production systems add cgroups, namespaces, and MAC, but all of that still rests on these fundamentals.

TL;DR

The OS is trusted firmware+software that multiplexes CPU, memory, and devices while enforcing isolation, user programs never touch privileged hardware directly, only through audited system calls.

Control flow across the boundary

Diagram: user code, syscall entry, kernel subsystems, drivers, and return.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
  U[User: libc + app] -->|syscall insn| K[Kernel: syscall entry]
  K --> V[VFS / MM / sched]
  V --> D[Drivers / IRQ]
  D -->|wake| K
  K -->|iret| U

Deeper narrative

Treat the OS as a policy engine wrapped around hardware mechanisms: the same MMU hardware can implement swapping or hard caps; the same timer interrupt can drive CFS fairness or RT deadlines. Interviewers reward connecting textbook layers (trap → syscall dispatch → return path) to observability (`strace`, `perf`, `/proc`).

Syscall path as a story

Ordered steps from intent in user space to return from the kernel.

User issues a capability

Libraries translate intent (`fopen`) into syscalls (`openat`) with validated pointers and flags.

Privileged transition

CPU raises privilege, kernel copies args, indexes syscall table, runs handler in process context.

Kernel cooperates with hardware

Handlers may sleep on wait queues; device interrupts complete I/O and wake waiters.

Return to user

Kernel packs errno + retval, restores user GPRs, lowers privilege; user code continues or handles errors.

Privilege ring sketch

Text diagram: user ring vs kernel ring and the syscall/fault crossing.

  USER                KERNEL
  ring 3              ring 0
    | syscall / fault |
    v-----------------^

Takeaways

Dual-mode operation

User mode cannot program the MMU, reprogram the timer chip, or disable interrupts. Illegal attempts cause faults; legal transitions use syscall/exception vectors into kernel mode.

Mechanism vs policy

A runqueue is a mechanism; CFS weights are policy. Exams often blur them, state clearly which layer you mean when answering design questions.

Boot is part of the OS story

Before the first user process runs, the kernel must set up page tables, trap vectors, physical memory maps from firmware, and early drivers, otherwise syscalls have nowhere to land.

Observability wins interviews

Pair every definition with `strace`, `/proc/[pid]/maps`, or `perf stat` so you can narrate a concrete path from syscall to driver.

Did you know?

Modern CPUs expose multiple privilege rings; mainstream OSes use two heavily (user/kernel) and leave others for hypervisors or firmware.

A single `read()` can block in the kernel without consuming a CPU core, threads wait on wait queues until the disk interrupt completes.

Linux exposes thousands of tunables under `/proc/sys`: they are policy knobs on top of the same core mechanisms you study in textbooks.

Common pitfalls

Confusing APIs with mechanisms

`printf` buffering lives in libc; the actual console path may involve `write`, tty discipline, and driver DMA, trace before claiming “one syscall.”

Ignoring firmware/boot

Secure Boot, ACPI tables, and IOMMU setup precede your first line of kernel C but shape security and NUMA topology.

What the OS is responsible for (sample mapping)

Resource → abstraction → typical mechanism.

What the OS is responsible for (non-exhaustive)

Resource	Abstraction	Typical mechanism
CPU	Threads / processes	Scheduler + timer interrupts
RAM	Private virtual address spaces	Paging + page tables + TLB
Disk / SSD	Named files & durability	VFS + block I/O + FS journal
Devices	File descriptors / IOCTL	Drivers + interrupt/DMA path

Glossary

Trap / exception

A synchronous control transfer to a kernel vector caused by an instruction (syscall, fault).

Mechanism vs policy

Mechanism is what is possible; policy chooses what should happen for fairness, security, or power.

ABI

Stable calling convention and syscall numbers user space relies on across kernel versions.

Interview follow-ups

Name something that is not a syscall but still requires kernel cooperation.
How does a hypervisor change the privilege story without changing user code?
Where would you look to prove a performance regression crosses the user/kernel boundary?

Callouts

Interview framing

Start answers with the invariant being protected (e.g., “TLB entries must match current address space”), then name the mechanism enforcing it.

Hands-on

Run `man 2 intro` on Linux for a categorized syscall list, then `strace -c true` to see how even trivial programs cross the boundary.

Walkthrough

60-second elevator pitch

Record yourself: multiplex CPU fairly, isolate memory with the MMU, persist with a file system, and expose capabilities through syscalls, then give one example of each from a web server stack.

Trace one syscall

Pick `openat`. Sketch user libc stub → `syscall` insn → kernel entry → path walk in VFS → permission check → inode allocation → fd table slot → return to user.

Deep dive

Traps, faults, and interrupts

A syscall is a synchronous trap from user code. A page fault is a fault that may be benign (demand zero) or fatal (SIGSEGV). Device interrupts are asynchronous, handlers run in kernel mode and may wake blocked threads.

Monolithic vs microkernel (tradeoffs)

Monoliths inline drivers for throughput; microkernels push drivers to user space and pay IPC costs for stronger isolation. Linux is monolithic with loadable modules; seL4 is a research microkernel exemplar.

Why virtual machines reappear in “OS” courses

A hypervisor is an OS for OSes: it schedules guest vCPUs, multiplexes physical memory with nested page tables (EPT/NPT), and virtualizes devices with virtio or device assignment.

See syscalls in flight

Terminal

strace -e trace=process,memory,file -o /tmp/trace.log your_binary
wc -l /tmp/trace.log

Self-check

Answer from memory first, then reveal explanations, same habit as System Design drills.

1Which action requires kernel mode on a typical Unix?

2A device interrupt handler usually runs in…

3The stable interface applications rely on across kernel versions is primarily…

4Which component typically runs first after power-on on a PC-class system?