When The Dashboard Flashes Red…

The push for project management efficiency often frames human effort as a bottleneck. With the advent of AI, the temptation to automate status reports, risk logs, and dependency tracking across complex programmes may same like an improvement. Yet, safety-critical engineering disciplines have long recognised a counterintuitive truth: deliberate inefficiency is not waste. The more advanced an automated system becomes, the more crucial a highly skilled human operator is when that system inevitably encounters an edge case.

The Lessons of High-Stakes Engineering

In 1983, cognitive psychologist Lisanne Bainbridge outlined the “Ironies of Automation” after studying the nuclear power industry. She noted that automation typically takes over the easiest, most routine tasks, leaving the human operator to act as a passive monitor. The paradox is that when the automation inevitably fails, it dumps the most complex, unpredictable scenarios onto a human who is out of practice and disconnected from the system’s current state. To mitigate this, the nuclear industry relies on “adaptive automation”—systems designed to deliberately hand control back to human operators to maintain their situational awareness and cognitive readiness. (See Case study 1 at the end of this post.)

The aviation industry faces the exact same challenge. Modern commercial aircraft can largely fly themselves, but aviation regulators have realised that an over-reliance on autopilot degrades fundamental flying skills. The US Federal Aviation Administration (FAA) issued formal guidance (SAFO 17007) explicitly encouraging airline pilots to routinely conduct manual flight operations. By manually managing pitch, energy, and environmental threats without algorithmic assistance, pilots maintain the perishable muscle memory and cognitive processing required to save the aircraft when instruments fail. (See Case study 2 at the end of this post.)

The Multi-Vendor Paradox

Project management is experiencing its own automation paradox, most visibly in complex, multi-vendor deliveries.

Imagine a major programme where Vendor A builds the software application, Vendor B provisions the infrastructure and Vendor C orchestrates the data pipeline. The interdependencies are dense, and the politics are fraught. Each vendor is inherently motivated to protect their profit margins, stick strictly to the letter of their Statement of Work, and swiftly avoid blame when integration inevitably stumbles.

It is highly tempting to use AI to bridge this gap—deploying tools to aggregate multiple Jira instances, auto-generate dependency maps, and forecast schedule impacts. But if a project manager automates their core governance tasks entirely, they become Bainbridge’s passive monitor.

When Vendor A’s software fails because of Vendor C’s undocumented data structures, the automated dashboard will simply flash red. A PM who has been entirely hands-off will lack the contextual intuition, the established vendor relationships, and the political capital to unravel the mess. They will be staring at a blank flight instrument panel, lacking the manual skills to land the plane.

Escaping the Indispensability Trap

There is a delicate balance to strike here. “Keeping your hands on the controls” is not an excuse to micromanage or hoard administrative tasks.

As discussed in my previous post on The Indispensability Trap, PMs who manually chase every ticket and act as the sole conduit for information build a cage of their own making. They become a single point of failure—indispensable for tactical execution, but unpromotable because the project cannot survive their absence.

The solution is not to manually do the work an AI could do in seconds. It is to automate the data while deliberately remaining hands-on with the friction. Your gut instinct in a multi-vendor project is not guesswork; it is a subconscious risk assessment built through continuous, tactile engagement with the people involved:

• Reading the room: Sensing the hesitation in a vendor’s voice during a steerco when discussing upcoming handover criteria.
• Probing the edges: Facilitating a messy, manual dependency workshop to force competing vendors to look each other in the eye, rather than relying on a tool to auto-generate a Gantt chart.
• Navigating the politics: Having unscripted conversations with a vendor’s lead engineer to uncover the truth about a delay before it gets sanitised by their account manager.

These manual interventions are technically inefficient. But by stepping back and letting an AI handle all the synthesis, the project manager loses their cognitive map of the programme’s dynamics, the hidden vendor agendas, and the system’s fragilities. You stop narrating tasks, and you start narrating decisions.

Balancing Automation and Hands-On Control

To manage multi-vendor projects safely, project managers must treat human-in-the-loop engagement as a mandatory safety mechanism, whilst building systems that scale beyond themselves.

The true value of a project manager in an AI-driven landscape is not their ability to act as a human router for data. It is their capacity to apply judgement, navigate ambiguity, and sense when the mechanics of a multi-vendor project are drifting out of alignment.

AI will undoubtedly consume the administrative heavy lifting of project management. Let’s use that automation to build scalable systems and free yourself from the indispensability trap. But we must consciously safeguard the tasks that force us to interact with the messy, human reality of our deliverables. Deliberate inefficiency guarantees that when the dashboard fails and the vendors start pointing fingers, we still know exactly how to fly the project.

In conclusion, AI may seem like something new and clever but the same design principles that have protected any engineering breakthrough needs to be a applied. There are important lessons to be learned from these incidents and understood in any new application you may be considering. Let’s learn from mistakes of other technology breakthroughs in automation and stand on the shoulders of giants by learning from their mistakes.

Case Study 1 : Three Mile Island incident.

The Three Mile Island (TMI) incident on March 28, 1979, was the most severe accident in U.S. commercial nuclear power plant history, resulting in a partial reactor core meltdown. While initiated by equipment failures such as a loss of main feedwater, the severity of the accident was primarily driven by human errors, design-related problems, and poor human-machine integration.

The Role of Automation and Interface Failures

The automated systems and control room interfaces actively misled the operators, significantly exacerbating the crisis:

• Misleading Indicators: A critical flaw involved the indicator light for the pilot-operated relief valve (PORV). When the automated control system sent a signal to close the valve, the indicator light turned off. However, this only confirmed that the “close” command was issued, not that the valve had physically closed. The valve had actually failed to close, allowing vital coolant to escape, but operators incorrectly assumed it was shut because of the misleading interface.
• Alarm Flooding: During the incident, approximately 750 different alarms sounded in the control room. Because these alarms largely looked and sounded identical, operators were overwhelmed by the flood of sensory information, making it impossible to quickly distinguish minor deviations from life-threatening failures.

Unfamiliarity and Human operator Errors

The operators were not reckless; they followed their training, but that training had not prepared them for this specific, anomalous scenario.

• Faulty Mental Models and Confirmation Bias: Operators were strictly trained to avoid overfilling the reactor. When gauges seemingly suggested water levels were rising, operators did what they were supposed to do and reduced the coolant flow, completely unaware that the system was actually losing water. Once this faulty mental model was formed, operators subconsciously filtered out new information that challenged it—a phenomenon known as confirmation bias.
• Event-Based Processing: At the time, plant procedures were “event-based,” meaning operators had to positively identify the specific event before they could select and implement a mitigation procedure. When faced with anomalous instrumentation readings, operators futilely attempted to pattern-match the situation to a known event, significantly delaying crucial corrective actions.

Lessons Learned

The TMI accident prompted sweeping, permanent changes in nuclear power plant operations, emergency response planning, and human factors engineering worldwide.

• Transition to Symptom-Based Procedures: The industry transitioned from event-based to “symptom-based” procedures. This allowed operators to respond directly to the physical symptoms of the plant (e.g., loss of coolant) without needing to immediately diagnose the exact root cause, a change operators later cited as one of the greatest improvements resulting from the accident.
• Human Factors in Design: The incident firmly established human performance as a critical component of plant safety. It highlighted the necessity for control interfaces to provide clear, direct feedback on the true physical status of components, rather than just the last commanded action. It also spurred improvements in control room ergonomics, meter markings, and color coding.
• Defense-in-Depth Validation: Despite the catastrophic operational and interface failures, the physical “defense-in-depth” design of the plant—specifically the sturdy containment buildings—successfully prevented the vast majority of radioactive materials from escaping into the environment, demonstrating the immense value of robust engineering margins.
• Enhanced Training and Oversight: The incident led to vastly increased requirements for reactor operator training and the introduction of “beyond-design-base-accident” instructions. It also accelerated the creation of international oversight systems, such as the International Atomic Energy Agency’s Operational Safety Review Team (OSART).

Case study 2: Air France Flight 447 accident

On June 1, 2009, Air France Flight 447, an Airbus A330 traveling from Rio de Janeiro to Paris, crashed into the Atlantic Ocean, resulting in the deaths of all 228 passengers and crew on board. The sequence of events was initiated when ice crystals obstructed the aircraft’s pitot tubes, which measure airspeed. This blockage caused inconsistent airspeed indications, prompting the autopilot and auto-thrust systems to automatically disengage. The flight crew reacted incorrectly to this failure, taking the aircraft outside of its safe operating limits and inducing an aerodynamic stall from which they never recovered.

The Role of Automation and Human Monitoring

The tragedy of Flight 447 highlighted critical flaws in how human operators interact with highly automated flight deck systems:

• The Passive Monitor Problem: Modern flight deck automation relies on crews to perform a passive monitoring role rather than actively operating the aircraft, which can lead to a drop in vigilance. When the autopilot disconnected, the pilots were abruptly forced to manually fly the aircraft at a high altitude—a task they were not accustomed to doing.
• Poor Human-Computer Interfaces: The systems provided inadequate and confusing feedback to the pilots. For example, the automated “flight director” gave faulty navigational instructions, and while the computers identified airspeed inconsistencies, they did not clearly display this information to the crew. Crucially, the aircraft’s actual “Angle of Attack” (the key metric for identifying a stall) was not directly displayed to the pilots.
• Over-reliance and Confusion: The pilots exhibited an over-reliance on the aircraft’s automation, failing to understand that the plane could stall once the automated flight protections were stripped away.
• Misleading Alarms: As the crew struggled amidst a barrage of alarms, the stall warning system behaved in a highly confusing manner. When one pilot correctly pushed the nose down to gain speed, valid data momentarily returned to the computers, which paradoxically caused the “stall” alarm to sound again—making the pilot mistakenly believe his correct action was making the situation worse.

Lessons Learned

The investigation into Flight 447 prompted the aviation industry to re-evaluate the relationship between pilots and automation:

• Prioritizing Manual Flying Skills: French authorities concluded that the pilots were not adequately trained to fly the aircraft manually during an equipment failure or high-altitude stall. The accident emphasized that pilots must maintain core manual flying competencies and not rely entirely on computers to interpret what the airplane is doing.
• Improving Automation Design: The disaster showed that automated systems must act as better “team members” by providing clear, future-oriented information. Interfaces need to highlight the origin of a problem rather than just displaying failure messages for the consequences.
• Understanding System Limits: The aviation community recognized that inadequate training regarding “alternate law” modes (where automated protections are reduced) leaves pilots prone to panic when an emergency exceeds their knowledge of the system.