System Design Security
Audit Logging for Security
Finance asks who exported 40,000 invoices at 02:14. Application debug logs rotated two hours later. The SIEM has HTTP 200s with no actor. That is not an observability gap; it is a missing security control.
A records actor, action, resource, and outcome so investigations, detections, and compliance have evidence. This note is not RED metrics or trace sampling: those belong in logging-and-monitoring. Here the log is an asset attackers will try to erase.
Evidence journey
Emit structured facts
Actor, action, resource, outcome, reason.
Ship without silent drop
Buffer, sequence, alert on lag.
WORM plus chained digests.
Rules, holds, dual control on delete.
What actually happens
On every security-relevant decision—allow, deny, grant, revoke, export, impersonate, break-glass—the control plane writes one structured event. Fields name the , action, resource, tenant, outcome, , policy version, UTC time, and . Client-supplied strings are fields, never new lines of text.
Events leave the host through a pipeline that can buffer, not through “best-effort stdout.” The durable copy lives in storage with a . A gets a searchable replica for . Deleting either copy is itself an audited, dual-controlled action.
Completeness includes denials and “nothing happened” admin views. Attackers probe with 403s; if you only log 200s, you lose the reconnaissance chapter.
Naive design, and the invariant that must hold
Naive claim
“info logs in the app cluster, 7-day retention, and we can grep Kubernetes if something looks weird.”
Counterexample
An insider with deploy rights truncates the log volume, disables the shipper “to save CPU,” and exports customer data. Seven days later legal asks for 90 days. Grep finds request logs with no actor and a username containing injected newlines that forged a fake admin trail.
Correctness invariant
What must never become false
Security-relevant actions produce an append-only event that outlives the producing host. Gaps, chain breaks, and pipeline disables are detected. Secrets never appear in fields. Retention and holds are enforced outside the app’s own credentials.
How to choose what to log
1. Would this answer “who touched this asset?” Authz decisions, grants, role changes, data exports, decryption of locked fields, and impersonation always qualify.
2. Can we store it safely? Prefer identifiers over payloads. Hash or tokenize account numbers. Never log tokens, passwords, or raw document bodies.
3. Who can destroy it? If the same role that administers the app can wipe SIEM, you have one control plane pretending to be two.
Simple rule of thumb
Structured, complete, tamper-evident, and separately administered. Sample debug logs; do not sample security audit events. Detect silence as loudly as you detect a mass export.
Minimum event schema
Read the diagram definition
flowchart LR
subgraph Required["Required fields"]
A[Actor plus impersonator]
B[Action]
C[Resource and tenant]
D[Outcome]
E[Reason code and policy version]
F[Time UTC plus correlation id]
end
subgraph Forbidden["Do not store"]
S[Secrets and tokens]
P[Raw PII beyond what investigation needs]
N[Unencoded user strings as whole lines]
end| Field | Why | Failure if missing |
|---|---|---|
| actor + impersonator | Who, and who was wearing whose badge | Support “as user” looks like the victim acted |
| action + resource + tenant | The triplet investigators query | Cannot scope a breach to one customer |
| outcome + reason code | Allow/deny and why, plus policy version | Cannot replay whether the control was correct |
| time UTC + correlation id | Order and stitch across hops | Clock skew and orphan gateway rows |
Encode against : structured JSON or protobuf, length limits, no raw control characters. Clock sync (NTP/PTP) is part of the schema: unsynchronized hosts scramble sequences even when ids are perfect.
Pipeline: no silent drops
Read the diagram definition
sequenceDiagram
autonumber
participant PEP as App or PEP
participant Buf as Local buffer
participant Pipe as Audit pipeline
participant Store as WORM plus hash chain
participant SIEM
participant Det as Detection
PEP->>Buf: Structured event: actor, action, resource, outcome
Buf->>Pipe: Batch with sequence numbers
Pipe->>Store: Append, commit digest
Store-->>Pipe: Chain head
Pipe->>SIEM: Copy for search and rules
SIEM->>Det: Rule match
Det-->>SIEM: Alert with correlation id
Note over Buf,Store: If pipeline lags, local buffer must not drop security events silentlyLocal buffers need a max and a policy: block the security-sensitive action, shed non-security logs first, or fail the request. Never “drop audit to protect p99 latency” without naming that as an accepted loss of evidence. Sequence numbers expose gaps; a should fire on unexplained jumps.
Dual-write to WORM and SIEM. SIEM indexes can be rebuilt; the append-only object store is the source of truth. Hash the batch, chain it, and copy the chain head to a second account.
Integrity: WORM and hash chains
Read the diagram definition
flowchart TD
E1[Event batch hour 01] --> H1[Digest H1]
E2[Event batch hour 02] --> H2["H2 = Hash(H1 || batch02)"]
E3[Event batch hour 03] --> H3["H3 = Hash(H2 || batch03)"]
H3 --> Anchor[Anchor chain head in second store]
Anchor --> Monitor[Gap and break alerts]
Monitor --> IR[Incident: treat missing hour as evidence loss]stops silent overwrite. A detects missing hours even if someone deletes objects and hopes nobody notices. Anchor the latest digest in a KMS-signed record or a second cloud account the app role cannot touch.
Administrators of the log plane need . Changing , disabling a source, or granting SIEM wipe must emit an audit event in a third place (ticketing plus a control-plane log). Watchers need watchers.
Detection, privacy, and retention
Useful starter rules: mass export above a baseline, grant of a privileged role, burst of denials for one actor then a success, impersonation without a ticket id, pipeline lag beyond budget, chain-head stall, sudden drop in event volume per service.
Privacy: collect the minimum that still answers investigations. Tenant and object ids usually suffice; message bodies do not. Mask in the producer, not only in the SIEM UI—replicas and support bundles leak. Access to audit search is itself audited.
Size to detection lag plus legal need. If you often notice incidents at day 40, 30-day hot storage is a policy bug. must pin objects by matter id so expiry jobs skip them. Prove both paths with drills.
Worked numbers
15k QPS of API traffic, 8% security-relevant (authz decisions, admin, exports) → 1,200 audit events/s. At 1.5 KB structured JSON, that is ~1.8 MB/s ≈ 150 GB/day before compression. Two replicas plus indexes can be 400–600 GB/day. Budget it as a product cost, not a side effect of “more logging.”
Buffer: 60 seconds at 1.8 MB/s ≈ 108 MB RAM per instance, or a local disk queue. If p99 pipeline latency is 400 ms and you alert at 30 s lag, you still have evidence during a SIEM blip—unless you drop.
Chain: hourly batches, 24 digests/day, trivial to verify. A missing hour at 1,200 events/s is 4.3 million events you cannot replay. That is the severity of a “quiet” shipper crash.
Trade-offs and failure modes
Completeness vs cost. Logging every packet is not a security audit. Logging none of the denials is not one either. Choose event classes with owners.
Fail closed vs availability. Blocking checkouts when the audit buffer is full protects evidence and hurts sales. Many teams fail closed only for admin and export, fail open for reads, and page immediately either way.
Hot search vs WORM. SIEM is convenient and mutable. WORM is awkward and trustworthy. Keep both; investigate from SIEM, prove from WORM.
PII vs reconstructability. Over-redaction makes the tape useless; under-redaction makes the tape a second breach. Prefer stable identifiers you can join under legal process.
Central pipeline vs app-local files. Central is searchable; local files vanish with the node. Local buffers are a shock absorber, not the archive.
Still fails when…
- Clock skew without monotonic sequence numbers.
- Sampling applied to audit events “like other logs.”
- The app role can DeleteObject on the WORM bucket.
- Impersonation logs only the target user.
What to measure
- Event volume vs request volume by class (detect silence).
- Pipeline lag, drop count, chain-head age.
- Time to retrieve a 40-day-old export event in a drill.
- Unauthorized SIEM queries by administrators.
Choosing and explaining security audit logging
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Audit logs are evidence of actor, action, resource, and outcome, stored so the producer cannot silently edit them.
Denials, grants, exports, and impersonation are first-class events with reason codes and correlation ids.
Detect gaps and chain breaks; treat silence as an incident.
When it belongs
Privileged actions, multi-tenant data, or regulated exports exist.
You need detection and investigation, not only application debugging.
Attackers or insiders could otherwise cover their tracks.
How it fails
Stdout logs rotated on the same host that was compromised.
Log injection from unsanitized fields.
Sampling or dropping audit events under load.
Retention shorter than detection lag, with no legal hold.
Trade-offs to weigh
Volume and SIEM cost versus completeness of denials and admin reads.
Fail-closed writes when the pipeline is down versus availability.
Hot searchable indexes versus append-only WORM proof.
Identifiers that protect privacy versus enough detail to investigate.
Common misunderstandings
Calling request-access logs a security audit trail.
Confusing this with metrics, traces, or SLO dashboards.
Storing tokens or document bodies “for completeness.”
Letting the production app role delete its own evidence.
Wrap-up
Design the audit path as a security system: schema, pipeline, integrity, detection, and administration. If the camera can be turned off by the thief, it was décor.
Primary sources
Key takeaway
Security audit logs are append-only evidence with a schema, a second trust domain, and detection for silence. Log who did what to which asset, including denials, and never let the producing app erase the tape.