System calls and APIs

Syscalls are the narrow waist of the trust boundary: small surface, huge responsibility. Performance engineering often reduces syscall count or batches operations; security engineering audits argument validation and object lifetime rules around each entry point.

Start here

A system call is how user code legally asks the kernel to do something privileged. If you are new: read the definition block, then the ABI / errno panel, those two ideas explain most “my program works on one machine only” stories.

System calls (precise beginner definition)

A system call (syscall) is a synchronous controlled transfer from user mode to kernel mode, identified by a syscall number and a set of arguments laid out according to the platform ABI (application binary interface). The kernel validates pointers and sizes, performs the operation (or returns an error), then returns to user space with a small integer result and/or side effects (memory written, file position advanced).

Libraries (libc, language runtimes) wrap raw syscalls with ergonomic APIs (fopen, read). Debugging crosses layers: a “file error” might be libc mapping a negative kernel code to errno.

By the end of this page you should be able to

State why the kernel must not blindly dereference user pointers.
Explain `openat` vs classic `open` from a time-of-check/time-of-use perspective.
Describe what EINTR means and when retry is appropriate.
Name one Linux optimization that reduces syscall overhead for time queries.

Argument passing & validation (theory)

Crossing the boundary is not free: every pointer is untrusted until checked.

Registers vs memory block

On many ABIs, the first few arguments live in registers; large or variadic calls may use a struct in memory. The kernel must read user memory with copy_from_user-style helpers that handle faults safely.

Confused deputy

If the kernel followed a user pointer into an object without checking ownership, attackers could trick it into acting on the wrong file or mapping, validation is a security property, not optional niceness.

Atomicity at the API

Some syscalls bundle check+use atomically (openat with a pinned directory fd) to avoid races that separate stat + open cannot fix.

Return paths

Fast paths may avoid blocking; slow paths may sleep on wait queues and reschedule. Returning to user space is also where pending signals may be delivered, hence EINTR semantics on blocking calls.

errno convention (conceptual)

Linux kernel internals often use negative error codes; libc wrappers translate to -1 return and set thread-local errno. When reading strace, know which layer you are interpreting.

TL;DR

A system call is a synchronous trap from user mode to a kernel handler with a stable ABI: arguments go in registers or a struct per ABI rules, the kernel validates pointers and copies data to prevent confused-deputy attacks, then returns an integer status or value.

Syscalls are the stable contract; everything else in the kernel can churn. Costs include marshalling, copying userspace buffers, security checks, and lock acquisition in the slow path-`vDSO` exists to avoid syscall overhead for time and getcpu on Linux.

Entry sequence (ordered)

From user stub to return to user space.

Step 1

User stub packages arguments

Registers or stack per ABI; errno handling conventions differ (`-errno` vs separate errno location).

Step 2

CPU enters kernel mode

`syscall/sysenter` on x86_64 loads STAR/SFMASK and jumps to `entry_SYSCALL_64`.

Step 3

Dispatcher validates

Pointer checks (`copy_from_user`), capability checks, and refcounting begin here.

Step 4

Return path

May reschedule if the thread blocked; restores flags and user stack, sets rax.

Core ideas

Validation is mandatory

The kernel must not dereference user pointers blindly-`copy_from_user` / `access_ok` style checks prevent crashes and privilege escalation.

Atomic syscalls reduce races

`openat` relative to a dirfd avoids TOCTOU path races that plain `open` had when directories moved between stat and open.

Syscall overhead matters at scale

High-QPS servers batch work, use `recvmmsg`, io_uring, or mmap rings to amortize boundary crossings.

Return conventions teach debugging

Negated errno on Linux libc wrappers vs raw negative kernel returns, know your layer when reading strace.

Did you know?

Linux `syscall(2)` numbers differ by architecture, cross-arch containers must match userland and kernel ABIs.
Many syscalls are restartable after signals (`SA_RESTART`) to avoid leaking EINTR handling bugs in old code.
`vdso` provides fast user-space helpers for `gettimeofday` without full syscall cost on some CPUs.

Security & correctness

TOCTOU on paths

Check-then-use across user/kernel boundary can race; use `openat` + `O_NOFOLLOW` patterns for privileged code.

Assuming syscalls are cheap

High-frequency `gettimeofday` historically hurt, vDSO mitigates; still profile.

Signals and blocking syscalls

When a signal arrives while you block inside read, you may see EINTR. Application code either retries carefully or relies on SA_RESTART semantics for certain calls, both are part of the syscall contract story.

Glossary

vDSO

Virtual dynamic shared object mapped into every process for fast userspace helpers (time, syscall avoidance).

errno

Thread-local error indicator in POSIX; kernel sets negative internal codes mapped by libc.

Restartable syscall

Some blocking syscalls adjust `-ERESTARTSYS` semantics after signals, kernel detail.

Drill questions

Why can `read` return fewer bytes than requested without being an error?

What is restartable atomic sequence handling for certain syscalls after signals?

How does `io_uring` change the syscall story?

Security

Kernel bugs in syscall paths are critical severity, patch quickly; assume attackers craft malicious argument combinations.

x86-64 syscall path (simplified)

User puts number in `rax`, args in `rdi,rsi,rdx,r10,r8,r9`, executes `syscall`; CPU loads `STAR/LSTAR/SFMASK` MSRs to jump to `entry_SYSCALL_64`, saves regs on the thread stack, dispatches via table.

Compare two APIs

Contrast `read` + loop vs `pread` for thread safety when multiple threads read the same fd at different offsets.

Count syscalls per request

Terminal

strace -c -e trace=%file,%desc curl -s https://example.com > /dev/null

Self-check

1Why does the kernel copy strings from user space instead of dereferencing them directly?

2`openat` primarily helps with…

3EINTR after a syscall means…

4A syscall handler must never…