Adding automation to critical systems tends to create new kinds of failure that are harder to catch than the ones it removes. 0.50 Agree Disagree

Argument TreesHow arguments are scored →

Weight for 0.00 · weight against 0.00 · net +0.00

Each reason is a claim with its own page. Score = sign x (2 x Truth - 1) x Conf x Link x Imp x Uniq, and every column is here so the row can be multiplied out and checked. A reason argued false scores negative and counts against the side it is filed on; a reason nobody has argued yet scores exactly 0, and so does one whose own page has no work behind it.

What Would Change the AnswerTruth scores →

Every claim beneath this page held at false and then at true, one at a time, with the whole graph recomputed each time. "If false" and "if true" are where this page’s truth score lands; "Move" is how far apart they are. "If settled" is that same distance with the input’s confidence held at 1, which is what it would be worth once the work behind it is finished, so the gap between Move and If settled is the value of doing that work. This is one input at a time and finds single points of failure; it does not test several inputs moving together, which is how correlated assumptions actually fail.

Related Beliefs

1. Self-driving cars can clearly be made safer than human drivers on ordinary roads. 0.50
2. Adding automation to critical systems tends to create new kinds of failure that are harder to catch than the ones it removes. (this page)

The other beliefs filed under Engineering; this one is listed unlinked.

Not filled in yet

Parts of the template nobody has filled in here. They are named rather than shown, because an empty table is not a finding, and each one is a reason the confidence above is not higher. A part that takes rows has a form for its first one.

What this page needs right now

Named gaps, read off the tables above, so a newcomer can see exactly where a contribution goes. None of them requires agreeing with the page, and each one links to the form it goes in.

What the numbers are made of

Share on the agree side

no scored weight either way

Confidence

32%. Weakest parts: scrutiny 0% · grounding 0%

Depended on

ReasonRank 0.0033, 50 of 2173

Strongest reason for

Automation hides what the system is doing, so when it fails the operators are surprised and have no time to understand what went wrong. +0.00

Strongest reason against

A machine's possible failures are written into its rules and can be inspected and tested before it ever runs, while a person's failures come from mood, fatigue and distraction that nobody can inspect in advance, so the new failures are easier to catch, not harder. -0.00

What would move this most

Settling Automation hides what the system is doing, so when it fails the opera…: 0.50 if false, 0.50 if true

Coverage

Of 6 scored rows, 6 have no linkage page, 6 have no importance page, 6 have no uniqueness page. This page reads 6 other pages. 6 of them are one-sided or empty: Automation hides what the system is doing, so when it fails the operators are surprised and have no time to understand what went wrong.; Operators who rarely act lose the skills they need in the moment the machine hands control back to them.; Automated parts are copied everywhere, so one hidden bug can fail in many places at once instead of one crew making one mistake.; A machine's possible failures are written into its rules and can be inspected and tested before it ever runs, while a person's failures come from mood, fatigue and distraction that nobody can inspect in advance, so the new failures are easier to catch, not harder.; Automated failures are logged and repeatable, so once they happen they can be found and fixed for good, while human lapses repeat forever.; The new failures automation creates can be designed against with alarms and backups, and every safety-critical industry has done so..

Bottom line

The disagreement is whether the surprising failures of automation are a permanent cost or a design problem that gets solved.

Structural Checks

Faults the shape of the argument can show without reading a word of it: a claim used to support itself, a conclusion listed among its own premises, the same page counted twice. None of these changes a score. The wiki also specifies pattern-matching the prose for ten named fallacies; that is deliberately not done here, because a matcher looking for “attacking the person rather than the argument” would fire on the central legitimate argument on this whole site, and a score moved by a regular expression has no page to appeal to.

Scoring EngineTruth scores →

Cite this page Adding automation to critical systems tends to create new kinds of failure that are harder to catch than the ones it removes. Idea Stock Exchange, belief page adding-automation-critical-systems, truth 0.50, confidence 32%, revision 42c8344 of 2026-10-04. The revision is what makes this quotable: these numbers move as the argument is worked on, and rebuilding that revision reproduces them exactly.

Every number on this page as data: adding-automation-critical-systems.json. Constants: K = 1 · UNARG = 0.5 · DEFLINK = 1 · DEFIMP = 0.5 · DEFUNIQ = 1. A reason placed under a conclusion is presumed relevant until a linkage page says otherwise; the burden is on the challenger. The neutral start for importance, the wiki's own rule.

Every value here is computed from the tables above at build time. Nothing is typed.