Falco + Nginx Plugin Development: Falcoya's Days 78-84

〜 The Hard Wall of Environment Differences and Startup Sequence 〜

Facing environment differences and startup sequence challenges

Looking Back

The previous period (Days 73-77) was a week where we gradually made "unstable stability" our own. We untangled timeouts, nginx installation issues, and dependency conflicts, transforming test results into team language with summary.html. By letting go of Docker in Docker and switching to Kubernetes-first, E2E finally approached a state of "flowing without stopping."

—— As an extension of that, this week we hit the hard wall of environment differences and startup sequence.

Day 78 (Oct 5) — Finalizing Kubernetes Support

Within Pods, systemctl is unavailable. Accepting this head-on, we consolidated to direct startup operations. We unified the startup method to nginx -g "daemon off;" and created a new script run-single-pattern-k8s.sh for environment auto-detection that switches between Kubernetes and standard environments. The goal was to "unify the startup method" to stabilize subsequent validation and diagnostics.

"The strongest approach is one that works the same way everywhere" — As TK said, just eliminating small branches reduced log variations and improved reproducibility.

Lesson

Unifying startup methods creates reproducibility. The importance of absorbing environment dependencies upfront.

Day 79 (Oct 6) — ENV-MIGRATE Expansion and Non-Privileged Design

ENV-MIGRATE-004/005. Assuming non-privileged containers, we added environment detection logic to each Category Runner (sqli / xss / cmd_injection / path_traversal / emerging_threats). Ports automatically switch from 80 to 8080, eliminating the need for root and avoiding conflicts. The purpose was to "absorb environment differences upfront" to eliminate downstream failures. As a result, we confirmed the same procedure works in Pods, locally, and on self-hosted runners.

TK: "Resilient systems are determined by 'initial assumptions.'" Just a few lines of branching eliminated many downstream errors.

Lesson

Non-privileged support is decided at the first branch. Environment adaptation design through port switching.

Day 80 (Oct 7) — TEST-VERIFY-001: Integration Verification

Reconfirmed end-to-end coordination between run-single-pattern-k8s.sh and Runners, aggregation via orchestrator. We acquired tests across three surfaces—local, Pod, and self-hosted—with the goal of guaranteeing that it "works with the same meaning regardless of environment."

Logs were quiet, recovery was stable. TK: "Quiet logs are a reward."

Lesson

Quiet tests are the greatest reward. The value of cross-environment operation guarantees.

Day 81 (Oct 8) — Documenting "Why It Works"

On this day, we added progress and implementation procedures to E2E_NGINX_MIGRATION_TASKS.md, and organized port constraints, log paths, and startup procedures in KUBERNETES_POD_COMPATIBILITY.md.

TK: "If you don't write down why it worked, your future self will struggle." We documented verification procedures and reproduction conditions with specific logs, turning reproducibility into specifications.

Lesson

Document why it worked to ensure reproducibility. Documentation as an investment in the future.

Day 82 (Oct 9) — PR #410 Update

This day was focused on reflecting the achievements of Kubernetes support and non-privileged container adaptation into PR #410. While carefully reviewing micro-differences like port switching within Pods, log path adjustments, and environment auto-detection logic, I picked up items pointed out during review.

The changes were small. However, to ensure that each modification wouldn't affect other environments, I needed to re-execute full runs each time. Local environment, Pod environment, self-hosted environment—running tests in each, checking logs, and confirming no unexpected side effects. It was a steady, time-consuming task, but I had learned from experience that cutting corners here would lead to major problems later.

TK: "Don't rush, first get everything through." Encouraged by those words, I carefully worked through each checklist item one by one. Small differences, but certain. That was the best approach I could take right now.

Lesson

Small differences, but certain. The importance of running everything through without rushing.

Day 83 (Oct 10) — Integration Test Re-run and Re-review Request

Added ENV-MIGRATE-004/005 fixes to the PR. Re-executed TEST-VERIFY-001, added results to PR comments, and requested re-confirmation. In terms of work status, uncompleted checklist items still remained, and work was continuing. The overall flow had come together, but I had a feeling that we were still "in the finishing phase."

TK: "Finishing takes time." I watched the logs while holding the reality of "not yet done."

Lesson

The finishing phase takes time. The courage to accept incompleteness and move forward.

Day 84 (Oct 11) — A154→A155: Why Configuration Correct but Doesn't Work

In the morning, PR #491 (Pattern #A154 fix) was merged. Environment detection logic was added to Phase 2's Normalization step, enabling log paths and port settings to switch according to the environment. It looked perfect as "environment support."

But the initial E2E test (Run #18429630180) was mercilessly stained red.

It failed at the Pre-flight check stage. No HTTP 200 returned, exit 1. I first suspected heredoc expansion and environment variable scope. But as I traced through the logs, the root of the problem was much deeper.

The nginx startup sequence.

The startup attempt within install-nginx.sh was misidentified as already "running," and the Normalization step side attempted a "reload," resulting in the remnants of the old process colliding with the new configuration. The nginx thought to be running was actually a "dying process."

Behind that collision, multiple layers were intertwined. Port inconsistency (80 ↔ 8080), configuration file generation order, and reload ambiguity.

In other words, while Pattern #A154 achieved "configuration correctness," another axis—"startup certainty"—was missing.

I recorded this as Pattern #A155. Created Issue #496 and added details to PROBLEM_PATTERNS.md. Lines 1088–1346 contain that analysis with five layers of causes. The bottom layer is "dual startup." The top layer is "reload ambiguity."

In other words, this failure wasn't a bug in one place, but showed that the entire design sequence had collapsed.

The correction policy was clear.

Start up only once. First remove startup attempts and operation checks from install-nginx.sh, and start up reliably only in the Normalization step. Before startup, safely stop with nginx -s quit, verify configuration with nginx -t, then start. After startup, confirm the actual port with ss -ltnp or netstat, and in Pre-flight check, output curl -v and nginx -T to leave diagnostic information.

TK said,

"'Configuration' and 'startup' are separate problems. Trying to solve it with one line of reload breaks the sequence."

Indeed, that was exactly right. Now that Pattern #A154 achieved environment support, what we needed to tackle was the design of "startup order."

By the end of the day's work, I had cut a branch:fix/pattern-a155-pre-flight-check-failure. Implementation is tomorrow. But today I finally understood "why it doesn't work." Just that alone felt like moving forward a little.

Lesson

Configuration and startup are separate problems. Start once, after configuration. The importance of design that doesn't break sequence.

Summary of Learnings

What I learned this week:

  • Absorb environment dependencies at the entry point (10/5)
  • Decide non-privileged and port switching at the first branch (10/6)
  • "Quiet tests" are the greatest reward (10/7)
  • Document why it worked to ensure reproducibility (10/8)
  • Small differences, but certain (10/9–10/10)
  • Configuration and startup are separate problems. Start once, after configuration (10/11 / A155)

Completed Tasks & Updated Documents

  • run-single-pattern-k8s.sh creation (environment auto-detection, Pod support)
  • ENV-MIGRATE-004/005 implementation (80↔8080 automatic switching, non-privileged support)
  • TEST-VERIFY-001 execution/re-execution (confirmation in each environment)
  • Document updates: E2E_NGINX_MIGRATION_TASKS.md, KUBERNETES_POD_COMPATIBILITY.md
  • PR #491 merged (A154: environment adaptation configuration)
  • PR #410 continued updates, re-review request (as of 10/10: work continuing)
  • Issue #496 creation (A155), PROBLEM_PATTERNS.md A155 addition (Lines 1088–1346)
  • Branch creation: fix/pattern-a155-pre-flight-check-failure (implementation to come)