[Cialug] Are they OK?

Wed Oct 6 00:04:22 UTC 2021

> During one of these routine maintenance jobs, a command was issued with
the intention to assess the availability of global backbone capacity, which
unintentionally took down all the connections in our backbone network,
effectively disconnecting Facebook data centers globally.
https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

> 1. Make each program do one thing well. To do a new job, build afresh
rather than complicate old programs by adding new "features".
https://en.wikipedia.org/wiki/Unix_philosophy

Brett Neese
563-210-3459

On Mon, Oct 4, 2021 at 7:44 PM Scott Yates <Scott at yatesframe.com> wrote:

> Even when it's not DNS....  It's DNS..  part deux:
>
> The working theory based on accounts from supposed employees is that the
> tools they use to push routing changes are also dependent on DNS and
> specific routes for auth and access, so the bgp config changes they pushed
> locked them out of being able to revert
>
>
> On Mon, Oct 4, 2021 at 7:40 PM David Champion <dchamp1337 at gmail.com>
> wrote:
>
> > Appears to be back. You may return to  your usual social media antics.
> >
> > -dc
> >
> > On Mon, Oct 4, 2021, 7:35 PM Brett Neese <brneese at brneese.com> wrote:
> >
> > > Return to monkee. It’s our only hope.
> > >
> > > On Mon, Oct 4, 2021 at 3:12 PM David Champion <dchamp1337 at gmail.com>
> > > wrote:
> > >
> > > > How about uucp bang path email addresses?
> > > >
> > > > -dc
> > > >
> > > >
> > > > On Mon, Oct 4, 2021 at 3:10 PM Brett Neese <brneese at brneese.com>
> > wrote:
> > > >
> > > > > I agree with @foone: DNS is a failed experiment we need to go back
> to
> > > the
> > > > > days when Elizabeth Feinler personally managed the list of
> computers
> > on
> > > > the
> > > > > internet.
> > > > >
> > > > > https://twitter.com/foone/status/1445111523186122755?s=21
> > > > >
> > > > > On Mon, Oct 4, 2021 at 2:35 PM <chris at bynw.com> wrote:
> > > > >
> > > > > > I saw this on Twitter. The source says it's from a deleted FB
> > > employee
> > > > > > reddit account.
> > > > > >
> > > > > > "As many of you know, DNS for FB services has been affected and
> > this
> > > is
> > > > > > likely a sympton of the actual issue, and that's that BGP peering
> > > with
> > > > > > Facebook peering routers has gone down , very likely due to a
> > > > > > configuration change that went into effect shorty before the
> > outages
> > > > > > happened.
> > > > > >
> > > > > > There are people now trying to gain access to the peering routers
> > to
> > > > > > implement fixes, but the people with physical access is separate
> > from
> > > > > > the people with knowledge of how to actually authenticate to the
> > > > systems
> > > > > > and people who know what to _actually do_, so there is now a
> > > logistical
> > > > > > thing with getting all that knowledge unified."
> > > > > >
> > > > > > chris
> > > > > >
> > > > > > On 2021-10-04 14:25, David Champion wrote:
> > > > > > > See, it's not always DNS... sometimes it's BGP. Which breaks
> the
> > > DNS.
> > > > > > > Damn,
> > > > > > > it's always DNS.
> > > > > > >
> > > > > > > -dc
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 4, 2021 at 2:06 PM Brett Neese <
> brneese at brneese.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> "Everything I do breaks"
> > > > > > >> <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://techcrunch.com/2012/09/11/the-ceo-of-facebook-still-codes-on-the-side-but-tends-to-break-things/
> > > > > > >> >-
> > > > > > >> Mark Zuckerberg, 2012
> > > > > > >>
> > > > > > >> This is why we shouldn't let Mark push to prod.
> > > > > > >>
> > > > > > >> Brett Neese
> > > > > > >> 563-210-3459
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Oct 4, 2021 at 1:28 PM Scott Yates <
> > Scott at yatesframe.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Ya, apparently some think it is a BGP routing issue, and
> they
> > > > can't
> > > > > > fix
> > > > > > >> it
> > > > > > >> > till they can physically access those servers.
> > > > > > >> >
> > > > > > >> > On Mon, Oct 4, 2021 at 1:26 PM kristau <
> > kristau at protonmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Looks like their .onion link is also down:
> > > > > > >> > >
> > > > > > >> > >
> > facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion
> > > > > > >> > >
> > > > > > >> > > So it might be more than "just DNS" as some have
> speculated?
> > > > > > >> > >
> > > > > > >> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > >> > >
> > > > > > >> > > On Monday, October 4th, 2021 at 1:09 PM, David Champion <
> > > > > > >> > > dchamp1337 at gmail.com> wrote:
> > > > > > >> > >
> > > > > > >> > > > Has anyone checked in with our friends those big
> warehouse
> > > > > lookin'
> > > > > > >> > > >
> > > > > > >> > > > buildings in Altoona to see if they're doing OK? Maybe
> we
> > > > should
> > > > > > >> send a
> > > > > > >> > > >
> > > > > > >> > > > care package with coffee and / or hard liquor?
> > > > > > >> > > >
> > > > > > >> > > > -dc
> > > > > > >> > > >
> > > > > > >> > > > Cialug mailing list
> > > > > > >> > > >
> > > > > > >> > > > Cialug at cialug.org
> > > > > > >> > > >
> > > > > > >> > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > >> > > _______________________________________________
> > > > > > >> > > Cialug mailing list
> > > > > > >> > > Cialug at cialug.org
> > > > > > >> > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > >> > >
> > > > > > >> > _______________________________________________
> > > > > > >> > Cialug mailing list
> > > > > > >> > Cialug at cialug.org
> > > > > > >> > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > >> >
> > > > > > >> _______________________________________________
> > > > > > >> Cialug mailing list
> > > > > > >> Cialug at cialug.org
> > > > > > >> https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > >>
> > > > > > > _______________________________________________
> > > > > > > Cialug mailing list
> > > > > > > Cialug at cialug.org
> > > > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > _______________________________________________
> > > > > > Cialug mailing list
> > > > > > Cialug at cialug.org
> > > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > >
> > > > > --
> > > > > Brett Neese
> > > > > 563-210-3459
> > > > > _______________________________________________
> > > > > Cialug mailing list
> > > > > Cialug at cialug.org
> > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > >
> > > > _______________________________________________
> > > > Cialug mailing list
> > > > Cialug at cialug.org
> > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > >
> > > --
> > > Brett Neese
> > > 563-210-3459
> > > _______________________________________________
> > > Cialug mailing list
> > > Cialug at cialug.org
> > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > >
> > _______________________________________________
> > Cialug mailing list
> > Cialug at cialug.org
> > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> >
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
>