🎉 Giảm 20% cho đơn hàng đầu tiên!✦🚚 Miễn phí vận chuyển cho đơn trên 500.000₫✦⚡ Flash Sale — Số lượng có hạn!✦🎁 Quà tặng kèm cho đơn từ 1.000.000₫✦🎉 Giảm 20% cho đơn hàng đầu tiên!✦🚚 Miễn phí vận chuyển cho đơn trên 500.000₫✦⚡ Flash Sale — Số lượng có hạn!✦🎁 Quà tặng kèm cho đơn từ 1.000.000₫✦
The server nobody dared to reboot for 7 years: binary deleted, process alive in RAM, and one wrong VLAN digit
Quay lại Blog

The server nobody dared to reboot for 7 years: binary deleted, process alive in RAM, and one wrong VLAN digit

15 tháng 11, 202612 phút đọcVũ Khôi
English
#English
Vũ Khôi

Vũ Khôi

Tác giả tại Wotech Electronics Blog

The story of the legacy01 server nobody dared to restart

The server nobody dared to reboot for 7 years: binary deleted, process alive in RAM, and one wrong VLAN digit

A virtual machine running CentOS 7, with just 2 CPUs and 4GB of RAM, had been up for 2,561 days — more than 7 years without a single restart — under an unwritten company-wide rule: never restart it, even though nobody remembered what service it ran. According to a post by Lữ Minh Khương in the VGU Tech Alumni group, sourced from the BugX channel, on a Friday afternoon at 4:47 PM the server lost network connectivity and dragged more than 50% of order requests into errors. The investigation that followed uncovered two hard-to-believe facts: the application's executable had been deleted from disk years earlier, and the real cause of the outage was a single mistyped VLAN digit.

Quick answer: According to the post, the virtual machine legacy01 (CentOS 7, 2 CPUs, 4GB RAM) ran continuously for 2,561 days under an unwritten rule of never restarting. On a Friday afternoon, a Live Migration at 4:44 PM misconfigured VLAN 26 as VLAN 261, cutting the machine off the network; a fallback branch in the Order Service code calling port 8443 on legacy01 with a 30-second timeout left more than 50% of order requests hanging and then failing. On inspection, the inventory-gateway process was still alive in RAM, but its executable at /opt/inventory/inventory-gateway had been deleted from disk years earlier (shown as deleted in /proc), with no systemd unit or startup script. The team rescued the binary from /proc, fixed the VLAN back to 26, and after more than 2,580 days, legacy01 was retired gracefully.

Live Migration is the mechanism for moving a running virtual machine between physical hosts without restarting it. A process running with a deleted binary is a state in which the Linux kernel keeps the inode and the executable data in memory as long as the process still holds an open file descriptor, even though the file name has been removed from its directory — but the moment the process stops or the machine reboots, that data is reclaimed for good.

What was legacy01, and why did the whole company forbid restarting it?

The main character was a virtual machine with hostname legacy01, running CentOS 7, with 2 CPU cores and 4GB of RAM, and an uptime of 2,561 days. When a new engineer, in his second week at the company, was reviewing hundreds of old VMs and asked whether it could be security-patched or switched off, a senior engineer named Mark immediately shut the idea down: don't touch that machine, never restart it — why, even Mark did not know.

For months afterwards, asking around the company produced no clear answer. Some guessed it ran an old reporting service, others said it was related to warehouse integrations, and some were convinced it had been replaced 3 years earlier and was now just idling. But any mention of switching it off made everyone anxious, so things carried on — the classic "legacy" server: the person who set it up had long left, documentation was zero, monitoring was minimal, and the only answer was ever "it's important, just leave it alone, don't touch it".

Chat with Mark: Don't restart it

What happened on Friday afternoon at 4:47 PM?

The incident struck on a Friday afternoon. At 4:47 PM the first alert fired: legacy01 host unreachable. Grafana showed all metrics had stopped 2 minutes earlier; pinging the IP and SSH both reported Connection timed out.

Before Mark could even be messaged, a wave of critical alerts followed: the Order Service error rate shot above 50%, the Checkout API went critical, and the order processing queue began to back up. For users, customers could still browse products, add items to the cart, and log in normally — but pressing the checkout button left the request hanging for exactly 30 seconds before returning an error.

Alerts: legacy01 Host Unreachable and Order Service error rate above 50%

The first instinct is usually to suspect the database or DNS, but PostgreSQL showed normal CPU, connection counts, and query speed. Opening the Order Service logs revealed thousands of Connection timed out errors all pointing to a single address: 10.20.4.17:8443 — legacy01 itself.

Why did orders hang for exactly 30 seconds?

Checking through the Hypervisor Console, the first surprise was that the VM was still Powered On, with light CPU and RAM load, no hardware errors, and uptime still showing 2561 days. The machine had not hung or lost power — it was alive but had completely lost connectivity with the outside network.

legacy01 alive but cut off from the rest of the network

Scanning the open ports, the engineer found a process listening on port 8443 called inventory-gateway. Searching the name across all internal documentation, GitHub, deployment repos, CI/CD pipelines, Ansible playbooks, and Docker turned up not a single line. Contacting Anna, a backend engineer, revealed: although most of the flow had moved to the new Inventory Service, the code still kept a fallback branch calling legacy01 directly to check and reserve stock for an old warehouse partner. The branch had sat there for years because nobody dared delete it, unsure whether the partner still used it — and because the request timeout was 30 seconds, when legacy01 went unreachable the entire order flow jammed in a chain reaction.

Order Service waiting 30 seconds on the unreachable inventory-gateway

Diagram: the fallback branch from Order Service to inventory-gateway on legacy01

The secret in /proc: the process lives, the file died years ago

The fastest way to save the system was to find the application's executable and configuration files on legacy01, copy them to another server, and start it there. But following the process ID to /opt/inventory/inventory-gateway, neither the directory nor the file existed on disk.

VM console: the process exists but the executable is gone from disk

Digging into /proc — where the Linux kernel maps information about running processes — the real reason behind the 7-year "no restart" curse came out: next to the path to the process's executable, Linux displayed a single word: (deleted).

The mechanism, as the post explains, is quite interesting: when a running program has its binary deleted from disk, the process does not die immediately. The Linux kernel only removes the file name link from the directory, but keeps the inode and the executable data in memory as long as the process still holds an open file descriptor. But the moment the process stops or the machine reboots, the kernel reclaims all of that data and the executable is gone forever. Someone had accidentally deleted the application directory years earlier, but the process launched in 2019 had quietly sat in RAM for 7 years, carrying the warehouse-check flow for the whole company.

By 5:15 PM, when someone suggested trying a reboot, Mark had to type in all caps on Slack to stop it. The first thing the engineer did was extract and back up the still-living binary from the process's /proc to a safe place, gathering all readable environment variables and configuration. Then, firing a request directly from localhost to port 8443 on legacy01 returned a valid response — proving the application itself worked perfectly and the problem was purely at the network layer.

Incident channel: DO NOT REBOOT THAT SERVER

The real culprit: VLAN 26 mistyped as VLAN 261

Checking the VM's network card: the interface was UP, the IP and routing table were correct, but the ARP table and pings to the Default Gateway showed no signal at all; a packet capture showed packets leaving the card with none coming back.

Opening the infrastructure change channel, the engineer found a message posted about 40 minutes earlier: "Scheduled VM migration completed successfully. No customer impact expected." Comparing the virtualization event history minute by minute:

Time Event
4:44 PM legacy01 was Live Migrated from the old Hypervisor to a new Hypervisor
4:45 PM The VM lost all network connectivity
4:47 PM Monitoring started raising red alerts

Timeline: 4:44 migrated, 4:45 network gone, 4:47 alerts

Robert, the engineer who ran the Live Migration, was initially surprised too, because the Port Group names on both Hypervisors were identical. But placing the two hosts' network configurations side by side, line by line, the real culprit appeared: on the source Hypervisor, the Port Group was attached to VLAN 26, while on the destination Hypervisor, whoever had set up the network earlier had mistyped it as VLAN 261. One extra digit was enough to push all of legacy01's traffic onto a completely different network segment after the migration.

Fortunately, fixing the VLAN ID on the Port Group could be applied immediately without restarting the VM. The moment it was updated back to VLAN 26, pings to the Default Gateway answered, connections to port 8443 flowed, the order queue drained, and by 5:38 PM all Production alerts turned green.

How was legacy01 retired, and what is legacy02?

The story did not end with the network fix. The person behind the no-restart rule was David — an engineer who had left 2 years earlier. The inventory-gateway process had never been configured in systemd or packaged into a container; David had typed the command to run it by hand in a terminal in 2019, intending to "run it temporarily and write a startup script later" — and that "later" stretched across 7 years.

On the following Monday, the engineer dug through old Git repositories and found a repo named warehouse-connector with its last commit from 2019. The entire README was just 4 lines, with a one-sentence deployment guide: "Copy binary to server and run". No build pipeline, no health check, no auto-restart policy.

From the recovered source code combined with the binary rescued from /proc, over the following weeks the team wrote a proper deployment pipeline, built 2 instances running in parallel on new infrastructure, attached full monitoring, rewrote the architecture documentation, and — most importantly — tested the restart scenario many times. After more than 2,580 days, legacy01 was finally given the shutdown command to retire gracefully without dropping a single order.

Two new inventory-gateway instances with monitoring, pipelines, and documentation

The small detail at the end is one every engineer finds both funny and familiar: the moment legacy01 was switched off, Mark messaged to ask whether he knew what legacy02 (also CentOS 7, 2 CPUs, 4GB RAM, uptime over 2,130 days) was doing — because its note also contained just one line: "Do not restart".

Lessons from the legacy01 story

Problem Lesson
Process hand-run since 2019, no systemd, no startup script Every service must have a documented, tested way to restart — nothing runs "temporarily" by hand
Binary deleted from disk with nobody knowing Anything running in production needs copies, a build pipeline, and documentation — no single file should decide survival
Fallback branch to the old system left in code for years Clean up dead code and old fallbacks; a 30-second timeout on a side branch can jam the whole main flow
One mistyped VLAN digit during migration Infrastructure changes must be compared configuration-by-configuration between source and destination; "no impact" notices need verification
Nobody knew what the server ran for years A "do not restart" note is no substitute for architecture documentation and clear ownership

Frequently asked questions

1. Why couldn't legacy01 be restarted? Because the inventory-gateway process's executable had been deleted from disk years earlier; the process only survived in RAM thanks to Linux's inode mechanism, and a reboot would have erased the file permanently.

2. What caused legacy01's network loss? A Live Migration at 4:44 PM moved the VM to a new Hypervisor, where the Port Group was misconfigured as VLAN 261 instead of VLAN 26, pushing all traffic onto a different network segment.

3. Why did orders hang for exactly 30 seconds? Because the Order Service code kept a fallback branch calling port 8443 on legacy01 to reserve stock, with a 30-second timeout — when the machine went unreachable, every order request waited the full 30 seconds before failing.

4. How was the application rescued with the file deleted? The team extracted the still-living binary from the running process's /proc to a safe place, along with all readable environment variables and configuration.

5. What happened to legacy01 in the end? After more than 2,580 days of continuous running, it was replaced by 2 new instances with a pipeline, monitoring, and full documentation, then shut down to retire without dropping any orders. Right afterwards, Mark mentioned legacy02 — a similar machine with a "Do not restart" note.

This article was rewritten from a post by Lữ Minh Khương in the VGU Tech Alumni group on 15 November 2026, sourced from the BugX channel ("We Had One Rule... NEVER Restart This Server"). The technical details, timeline, and figures follow the original post.