Security Operations and Incident Response: A Hands-On Walkthrough

Security Operations and Incident Response: A Hands-On Walkthrough

Security operations isn’t a single tool or checklist — it’s a layered process spanning detection, response, and evidence preservation. This post walks through five operational areas with runnable code:

  1. Penetration testing scope definition and vulnerability scanning methodology
  2. Intrusion Detection Systems (IDS) versus Intrusion Prevention Systems (IPS)
  3. The six-phase incident response lifecycle (NIST SP 800-61 Rev. 2)
  4. Digital forensics principles for evidence preservation and chain of custody

Each section has its own code block, a live recording, and the reasoning behind why each piece matters.

Defining Scope in Penetration Testing

Every engagement starts with scope — without it, you don’t know what’s fair game. The scope defines which IP ranges are in-scope, which are explicitly excluded, and which assets might be honeypots (decoys designed to detect unauthorized scanning).

A scope definition has three parts: CIDR blocks, asset inventory with scan type, and verification logic that checks whether a target IP falls within the defined ranges. Here’s how it works in practice:

class ScanType(Enum):
    UNAUTHENTICATED = "unauthenticated"  # external-facing only
    AUTHENTICATED = "authenticated"      # internal agents / API creds

In-scope assets get authenticated scans (deeper, requires credentials); out-of-scope and decoy targets are scanned unauthenticated or not at all.

The Code

The first script handles both scope definition and a simulated vulnerability scan against a web application configuration file:

What the Output Shows

The scope section defines three CIDRs: two /24 production networks (256 addresses each) and one /28 DMZ (16 addresses). The asset roster marks five hosts — three in-scope, one out-of-scope, one decoy.

The scope verification test confirms:

  • 10.0.1.50 → IN-SCOPE (within the first /24)
  • 198.51.100.50 → OUT-OF-SCOPE (outside all defined ranges)
  • 10.0.3.99 → OUT-OF-SCOPE (the decoy network is not in scope)
  • 203.0.113.1 → IN-SCOPE (within the DMZ /28)

The vulnerability scanner then examines a simulated application config and finds 4 findings:

  • 1 Critical: debug mode enabled (DEBUG = True), exposing stack traces and internal paths
  • 3 High: hardcoded secrets — SECRET_KEY, DB_PASSWORD, and API_KEY all found in plaintext

The severity tiering matters: a critical finding (debug mode) tells you something about access, while high findings (hardcoded creds) tell you about data exposure. A real engagement would correlate these against CVE databases for accuracy.

IDS vs IPS: Detection and Prevention Models

Both IDS and IPS use the same core technique — signature-based matching — but differ in what they do after a match:

  • IDS (Intrusion Detection System): monitors traffic, generates alerts. Passive. The traffic flows through unaffected.
  • IPS (Intrusion Prevention System): sits inline with traffic flow. When a signature matches, it can drop or block the packet in real time.

The detection logic is identical between them — what changes is the action model.

What the Output Shows

The script simulates seven network events: three benign (standard web requests) and four malicious (SQL injection, directory traversal, C2 beacon, port scan). The signature database contains pattern keywords for each attack type.

IDS mode generates 4 alerts — each malicious event triggers a signature match and an alert is generated. Benign traffic passes through without any notification.

IPS mode generates the same 4 matches, but reports them as BLOCKED rather than ALERT. The traffic that would have reached the application is dropped at the detection layer.

The key insight from this run: both systems produced exactly 4 detections from 7 events (3 benign passed through cleanly). The difference isn’t what they detect — it’s whether malicious traffic reaches its destination. This is why IPS deployment requires more tuning; blocking false positives directly impacts business operations, while alerting on them does not.

The Six-Phase Incident Response Lifecycle

NIST SP 800-61 Rev. 2 defines the incident response process in six phases:

  1. Preparation — tools, playbooks, team readiness (already done before an incident occurs)
  2. Identification — detecting and classifying the incident
  3. Containment — isolating affected systems (short-term + long-term)
  4. Eradication — removing attacker artifacts and root cause
  5. Recovery — restoring services to normal operation
  6. Lessons Learned — after-action review and process improvement

What the Output Shows

The simulation models a P1 (critical) web app compromise where an attacker exploited SQL injection, gained shell access, and initiated data exfiltration.

Phase-by-phase, here’s what ran:

  • Preparation (4 actions, 0 min): SIEM rules, IR playbook, evidence tools, backup verification — all completed before the incident. This phase is invisible during an actual event; its value shows up when things go wrong.
  • Identification (4 actions, 11 min): SIEM alert → log correlation → confirmed exploit → severity classification. Total detection time: ~11 minutes of focused analyst work.
  • Containment short-term (3 actions, 6 min): Network isolation, perimeter firewall block, egress ACL block. Fast action to stop the bleed.
  • Containment long-term (2 actions, 40 min estimated): 1 done (isolation VLAN), 1 pending (clean deployment).
  • Eradication (4 actions, 70 min estimated): Disk imaging and backdoor removal complete; patching and WAF rule review still pending.
  • Recovery (3 actions, 255 min estimated): Database restore, hardened app deployment, 72-hour monitoring — all pending.
  • Lessons Learned (3 actions, 210 min estimated): Review meeting, playbook update, automated evidence trigger — all pending.

The metrics section reveals a crucial distinction: the total action duration (sum of all task estimates) is 592 minutes (~10 hours), but the clock duration from start to finish is only 360 minutes (6 hours). The difference represents idle and waiting time — coordination gaps, approval delays, and handoff pauses that don’t show up in action items but extend real-world incident timelines.

Digital Forensics: Evidence Preservation & Chain of Custody

Digital forensics rests on a single requirement: evidence integrity must be provable at every point of transfer. This is where cryptographic hashing (SHA-256) becomes non-negotiable.

When evidence changes hands — analyst to IR lead, forensic examiner to investigator — the receiving party hashes the received data and compares it against the expected hash. If they match, the chain is intact. If they don’t, you have a broken chain and compromised evidence.

What the Output Shows

Three pieces of evidence are collected: a disk image, application logs, and a memory dump. Each gets an initial SHA-256 hash at collection time — that’s the “expected” hash for every subsequent transfer.

The disk image goes through three custody transfers:

  1. SOC Analyst → IR Lead: VALID (hash e19641c2…)
  2. IR Lead → Forensic Examiner: VALID (hash unchanged)
  3. Forensic Examiner → Lead Investigator: BROKEN — between steps 2 and 3, the disk image was tampered with. The expected hash (e19641c2…) no longer matches the received hash (ae90dfe3…).

The log file and memory dump each go through two transfers without any modification — both chains remain VALID.

Chain of custody summary: 5 out of 6 verifications passed. The one broken chain is at transfer Forensic Examiner → Lead Investigator for the disk image.

This run demonstrates why chain of custody isn’t optional. In the scenario above, once the hash mismatches at step 3, evidence integrity cannot be guaranteed past that point. Any analysis based on the tampered disk image — malware samples, log entries, file timestamps — becomes legally inadmissible because the provenance is broken.

Takeaway

Security operations is a chain of interlocking processes: scope definition prevents overreach and wasted effort, vulnerability scanning finds exploitable misconfigurations before attackers do, IDS/IPS provides detection with different operational trade-offs, incident response structures chaos into phases, and forensic chain-of-custody preserves the integrity of every piece of evidence. The common thread is verification — proving at each step that nothing changed without authorization.