A Dead Fuel Pump, 40 Miles from Nowhere (March 2024)
On March 14, 2024, I was 40 minutes from a vendor audit when my car died at a red light. No warning. Just silence.
I knew the symptom—crank, no start—but not the cause. I pulled onto the shoulder and Googled "how to start a car with a bad fuel pump" while trucks blew past me at 70 mph. The tricks looked simple enough: cycle the ignition a few times, bang on the tank with something heavy. I tried both. The car started, ran for two miles, then quit again.
Honestly, I'm not sure why the banging worked the second time. My best guess is it jolted the pump enough to catch. That's not a fix. That's a delay.
I called a tow. $150. The fuel pump replacement ran $480. The missed audit cost nothing—I called ahead, rescheduled for the following week. That time, the delay was cheap.
The Same Pattern, $52,000 Later
Four weeks later, I watched the same pattern play out on our production floor.
Our Allen-Bradley PLC-5 system—commissioned in 1998, patched more times than I can count—threw a fault during a peak run. We had redundancy on paper. In practice, we had a technician cycling power and hoping, same as I'd done on the shoulder of I-70.
That's when I stopped treating our automation budget like a price list and started treating it like an uptime guarantee.
The Quote That Looked Expensive Until It Didn't
Over the past six years, I've tracked $180,000 in cumulative automation spending across 30+ vendors. Every order goes into our cost tracking system—invoices, lead times, service callouts, the works. That spreadsheet is the only reason I trust my own instincts anymore.
In Q2 2024, we ran a formal comparison: refurbish the existing Allen-Bradley PLC-5 fleet or migrate to the CompactLogix platform. The rebuild quote came in at $38,000. The migration quote was $52,000, including commissioning and a two-day cutover window. Three vendors quoted the migration scope; prices ranged from $48,000 to $67,000 for the same line.
The $14,000 gap felt significant. Then I calculated the cost of one unplanned outage: $8,200 in lost throughput. We'd had three in the previous six months. The "cheap" rebuild option suddenly looked expensive.
Calculated the worst case: another six months of patch-and-pray, with a 30% chance of a major outage during peak season. Best case: smooth migration and lower maintenance. The expected value favored migration, but the downside—a failed cutover during Q4—felt catastrophic. We scheduled the cutover for August, our slowest month.
The 1000-Watt Mistake
Here's the surprise that flipped our timeline. In July 2024, a grid fault knocked out our panel for 90 minutes. We scrambled.
Someone wheeled in a Honda 1000 inverter generator—a 1000 watt inverter unit we usually keep for remote diagnostics and small field jobs. We figured it could run the HMI and one I/O rack. It couldn't.
Startup surge alone pulled more than the generator's continuous rating. The breaker tripped twice. We ended up renting a 12 kW diesel unit on a four-hour emergency callout. Rush rate: $650, including delivery and fuel.
The surprise wasn't the generator failing. It was realizing our "backup plan" was a guess. A 1000 watt inverter generator is fine for a laptop and a test meter. It is not a production continuity plan.
Buying the Guarantee, Not the Logo
After comparing those eight vendors over three months using our TCO spreadsheet, we chose to migrate. Not because the hardware was the cheapest—it wasn't. Not because the brand was familiar—though that helped with internal buy-in. We chose the quote that came with a written delivery guarantee and a training package.
The Allen Bradley PLC programming certification course added $4,200 to the contract. Our CFO asked why we couldn't just learn from YouTube. Fair question.
Here's the difference: a certified trainer walks your team through real fault scenarios on actual hardware. No guessing, no "I think this should work." Our lead technician passed the assessment in October 2024. The first production changeover he wrote solo saved us a $900 field service call. That's 21% of the training cost. Not a home run. But the real win came in December, when a legacy Allen-Bradley PLC-5 card failed during a weekend run—he swapped the migrated logic in two hours because he understood the architecture, not just the syntax.
During negotiations, I pushed for a penalty clause on the delivery guarantee. The vendor countered with a 2% price increase—about $1,040. I took it. That clause changed how their project manager treated our schedule. No "we'll get to it when we get to it." They hit the window. But if they hadn't, we'd have been credited $500 per day.
According to Rockwell Automation's lifecycle documentation (rockwellautomation.com, as of 2024), the PLC-5 family is classified as mature, meaning limited new feature development and shrinking spare-parts availability. Vendor reps confirmed lead times for some PLC-5 cards have stretched to 8–12 weeks. That's not a risk you want to discover during a shutdown.
Three Numbers I Track Now
I keep a small dashboard. It has three columns that didn't exist last year:
- Spare-parts lead time: measured in weeks, not "in stock." For PLC-5 cards, currently 8–12 weeks.
- Field service response guarantee: contract language, not the sales rep's promise. Ours says four hours for critical faults, with penalties.
- Actual generator startup surge: measured, not nameplate. Our critical panel needs 3.2 kW at startup, which is why the 1000 watt inverter generator was never going to work.
I've never fully understood why some vendors quote 10-day lead times and deliver in six, while others quote five and deliver in nine. My best guess is internal buffer practices—some companies pad their estimates, others pad their promises. But I know this much: when the line is down, "probably tomorrow" costs more than "guaranteed Friday."
What I mean is that the sticker price of a PLC—or a generator, or a fuel pump—is the least interesting number in the quote. The interesting numbers are the ones that show up six months later, when something fails and you're calculating how much the delay costs per hour.
That's the premium I'm willing to pay. Not for speed. For certainty.
The fuel pump cost me $480 to replace, plus $150 for the tow. I could have bought a $90 aftermarket pump and done it myself—I watched the mechanic do it in 40 minutes. But I didn't have 40 minutes that week. I had a vendor audit and a rescheduled meeting and a production line that was about to teach me a much more expensive version of the same lesson.
Next time, I'll pay for the guarantee up front. Not because I like spending money. Because I've seen what guessing costs.