Google Public CA is down(status.pki.goog)
241 points by aloknnikhil 7 hours ago | 26 comments
TMEHpodcast 6 hours ago
It is a well-known fact that the moment YouTube goes down, the collective productivity of Earth increases by approximately 4,000%, which is immediately squandered by everyone going to Hacker News to read comments about YouTube being down. I myself have taken to podcasts… an ancient medium in which people simply talk at you for ninety minutes without a single sponsorship for a mobile game, and this is considered a failure
PostOnce 5 hours ago
They've begun injecting obnoxious ads into the downloadable mp3s on a lot of podcasts I've found. Hyperlocal ads for tire shops and bakeries.

I don't want to buy tires, I want to learn about ______. The ads don't even make sense because they're irrelevant.

ideasarecool 40 minutes ago
At least it is somewhat relevant. Hearing ads about Irish telecom operator ads at the other side of europe is pretty goofy. What's the actual point? Just worsening the podcast experience?
0_____0 3 hours ago
VPN to Sweden to get the IP geolocated ads to retarget. The ads still exist but they're less obnoxious, and they're often in Swedish so you don't have to know what they're on about anyway.
VorpalWay 1 hour ago
And what are we Swedes supposed to do?
marak830 2 hours ago
Careful, I enjoyed this bonus (being in Japan and not being able to keep up with the ads)... so much so, that I started ignoring the Japanese. Including my wife. You can imagine how well that went.
moregrist 4 hours ago
Welcome to radio 2.0.

Give it another 10-20 years and your 2 hour podcasts will be 30 minutes of morning zoo DJ banter, 10 minutes of guests, and 1.5 hours of ads.

We’ll have reached peak 90s all over again. With any luck we’ll avoid recreating the conditions for another Nickelback and can stay in the weird zone where Trip Hop and pop punk could chart at the same time.

1313ed01 54 minutes ago
The 00's podcasts I listened to were often in 2-3 hour episodes, rarely well scripted (or scripted at all?), but a lot of fun and very amateurish. I re-listened to several entire series recently and the episode lengths were the only thing I think was worse than in newer podcasts.

On the other hand, if ads etc gets too annoying, I already have run all my downloaded podcasts through whisper to get transcripts with timestamps. Running some LLM to find ranges to delete would probably be quite easy. As a bonus I would be happy to also cut out all the filler repetitions that seem popular these days ("yes, X, I absolutely agree, [repeats everything X just said]"). Could probably cut 1 hour episodes to 20 minutes without losing any content.

blackoil 2 hours ago
> 2 hour podcasts

You have high hopes. Next YT tool will be to split anything long in 30s reels as brains will be completely incapable of focusing for longer.

SchemaLoad 4 hours ago
And it will all be AI generated specifically for you live.
staticassertion 5 hours ago
I listen to multi-hour unsponsored content on Youtube almost exclusively.
bdavbdav 52 minutes ago
This comment sponsored by Vivo barefoot. I really do wear them myself. Honest.
14 6 hours ago
Well one must also argue the opposite. I myself have gained immense knowledge from YouTube. I have learned things like phone screen replacements or phone battery replacements. I call myself a mechanic from the school of YouTube and have saved myself at minimum $10k in repairs doing the work myself. I have learned to make endless food recipes or create things like giant bubbles or slime for my kids. My point is that I bet sure for some YouTube is a massive time sink waste of time. But I also wonder how much it has improved the knowledge, skills and ability of others. My dad often mentions how had he had YouTube when he was younger how much it would have done for him. He talks about having to go to the library and if lucky there was a book that could show you the knowledge you were looking for. He says but now you can find not just the knowledge but for example specific knowledge like car make model and year and how exactly to do job xyz. Ultimately I just can not imagine life without the wealth of knowledge YouTube has given me.
TMEHpodcast 6 hours ago
Congratulations! You’ve successfully avoided YouTube Shorts.
com2kid 5 hours ago
YT shorts are up to 3 minutes now.

At this point it is just YT Vertical Videos.

marcosdumay 5 hours ago
Personally, I just scroll through them. They break the feed into well defined "chapters" at the end of what I can decide to look into the next one or go somewhere else because there's nothing good there today.

Also there's this woman that makes very funny shorts about software development and good long videos that aren't as good. I look for her shorts too.

jader201 5 hours ago
I just stay on my subscriptions page. Most of them don’t do Shorts, and the few that do don’t do many so they’re easy to ignore.
14 5 hours ago
Lol I laughed out loud reading this comment. When shorts first came out they annoyed me to no end. I searched for how to block them through settings or other ways to just make them go away.

But now days I can admit there are a few, very few, content creators who create shorts that are very informative and straight to the point that can cover a topic and give you many facts and let you decide if you want to seek more. Sometimes it is nice to have the 30 seconds Coles notes verses a video stretched out to 10 minutes to be eligible for monetization.

BUT, and this is a big but, the shorts and similar video platform trends scare me as a parent. I can see how my kids find a 1.5 hour movie boring but can scroll endlessly through shorts. It might seem harmless letting your kid just scroll on YouTube from my perspective is like an addiction and kids are getting that dopamine hit watching a clip and seconds later watching something else. I've learned that it is very important to be aware of what your kids are being accustomed to and push them in the right direction.

2Gkashmiri 1 hour ago
I watched a movie, same late night talk show host, something like "welcome night owls".

I "loved" the style but I haven't found any actual radio on the internet of that style or a podcast. Not sure about name of movie but I do remember it being in the last 10-15 years.

kyledrake 3 hours ago
People went ballistic on me a few months ago for bringing this up, but this is exactly the kind of outage that makes me really, really worried about extremely short lived certificates. https://news.ycombinator.com/item?id=46118371
codys 2 hours ago
I'm not sure I follow. This outage seems like it occurred for less than 1 day. The post you link to is about having certificates expire after 45 days. What's the connection you see?
jeroenhd 17 minutes ago
Some CAs are experimenting with shorter, 7 day certificates as well.

still not an outage that would endanger anyone's ability to renew in time, but for small or extremely shitty CAs (and there are a lot of those) such an outage may take enough time to cause issues in theory I guess?

TwoNineFive 1 hour ago
You didn't read it or understand it.
aaomidi 57 minutes ago
You know there’s more than one CA?
TwoNineFive 1 hour ago
Your license to website has been revoked.
kidfiji 6 hours ago
Ah, so that’s probably why YouTube is also down (at the time of this comment)
gzread 6 hours ago
I am playing a YouTube video (since the time of this comment) and it has not been interrupted.
brikym 6 hours ago
I am too. But I just loaded up a new youtube page and it's completely white except for a few menu buttons.
gilgoomesh 6 hours ago
It seems to be back, now.
dyauspitr 6 hours ago
You can still see your subscription videos, just not the homepage.
cvhc 6 hours ago
Searching also works. Actually it seems only the recommendation system is down, which I'd say isn't completely a bad thing.
tzs 6 hours ago
It is pretty annoying for those of us for whom the recommendation system actually works well.
6510 5 hours ago
What do you recommend?

(i'm that old)

LeoPanthera 6 hours ago
My subscriptions page just shows an error. And the app version won't load at all.
GeekyBear 6 hours ago
I'm able to play videos that are bookmarked in my browser, but the YouTube home page errors out.
cookiengineer 4 hours ago
> I am playing a YouTube video (since the time of this comment) and it has not been interrupted.

So you're using snakeoil certificates and MITM proxies at work?

ekr____ 6 hours ago
Perhaps the same underlying cause, but there's no reason why Google's public CA being temporarily down would bring YouTube down.
silverquiet 6 hours ago
If multiple services are affected, it's probably some underlying infrastructure issue.
thayne 5 hours ago
It could prevent Google from rotating in new instances, because they aren't able to obtain a certificate.

Although, if that is the case, I would expect to to impact basically every google site.

qmarchi 6 hours ago
Google uses mTLS for communications between systems and it could just be bad timing.
LPisGood 5 hours ago
Yeah companies which also operate CAs can print as many certs as they want so it’s tempting to use a bunch everywhere with very short expiry.
bathtub365 6 hours ago
The status history on the page makes it seem like this was intentional?

> 17 Feb 2026 11:32 PST A rollout is going to prevent issuance from occurring. We will provide an estimate on when issuance will stop.

> 17 Feb 2026 12:14 PST Issuance is beginning to stop. A fix to resolve the issue will roll out in about 8 hours

agwa 5 hours ago
This usually indicates that the CA was issuing non-compliant certificates and needed to prevent further non-compliance. Will be interesting to watch Bugzilla for the incident report: https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Progra...
nickysielicki 5 hours ago
What qualifies as a non-compliant certificate?
agwa 5 hours ago
It doesn't comply with one or more root store policies (which all incorporate the Baseline Requirements by reference, which incorporate various specs, such as RFC5280, by reference).

Mozilla root store policy: https://www.mozilla.org/en-US/about/governance/policies/secu...

Chrome root store policy: https://googlechrome.github.io/chromerootprogram/

Apple root store policy: https://www.apple.com/certificateauthority/ca_program.html

Baseline Requirements: https://github.com/cabforum/servercert/blob/main/docs/BR.md

There are countless examples of non-compliant certificates documented in the Bugzilla component I linked above. A recent example: a certificate which was backdated by more than 48 hours, in violation of section 7.1.2.7 of the Baseline Requirements: https://bugzilla.mozilla.org/show_bug.cgi?id=2016672

zerocrates 5 hours ago
The heading above that:

"There is an ongoing incident that will force issuance to be halted."

Feels like they were alerted to some current problem severe enough that "turn it off now" was the right move. Breaking the baseline requirements somehow maybe?

jtokoph 6 hours ago
> A fix to resolve the issue will roll out in about 8 hours

oof

catsquirrel28 6 hours ago
I guess it's good Google hasn't succeeded in forcing people to renew certificates every 8 hours (yet)
bawolff 4 hours ago
In theory 8 hours of downtime should be fine for a CA. Obviously not ideal, but the pki system is not meant to be a live system.
SchemaLoad 4 hours ago
Fairly sure it used to be pretty much a manual process where someone had to actually process your request for a certificate on the other side.
altairprime 6 hours ago
That feeling when you have to suspend production service until the time lock safe can be opened.
altairprime 2 hours ago
That feeling when you finally get the timelock safe open and have to do certificate work that shatters YouTube’s connection to the account personalization systems.
themafia 3 hours ago
The same amount of time it feels like it takes for my google functions to deploy.
tokyobreakfast 6 hours ago
It's a good thing we have ever-shrinking certificate lifetimes and automation never breaks. That's what I've been told, anyway.
bigbuppo 6 hours ago
Yeah, this could end up as the actual root cause of The Great Oops that I've been raving about for years. And Google probably would be the right company to fuck it up in the worst way possible since Google Knows Best In All Situations.
tokyobreakfast 6 hours ago
I don't subscribe to your newsletter. What about the Oops?
stickynotememo 1 hour ago
Do you have a blog post on the oops? I'd love to read it.
ocdtrekkie 6 hours ago
I can't wait for the Great Oops.
LPisGood 6 hours ago
Please tell me more about The Great Oops
bigbuppo 6 hours ago
It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data.

It will forever be known as The Great Oops.

Arainach 6 hours ago
It's not inevitable, it's essentially impossible.

There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".

rossjudson 3 hours ago
Delete a decryption key. Good luck! I'll see you at the end of time.

Break your control plane, and you can't stop the propagation of poison.

Propagate the wrong trust bundle... everywhere.

Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.

bigbuppo 2 hours ago
Children of the kubernetic line.
GeekyBear 3 hours ago
Google accidentally deleted customer location history data from customer devices (after intentionally deleting it from Google servers) just last year.

If didn't back it up yourself, it is gone forever.

ocdtrekkie 5 hours ago
Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it.

The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero.

Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blan...

Arainach 5 hours ago
.....no?

"We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes.

No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?

ocdtrekkie 4 hours ago
It's pretty much half the puzzle actually.

You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner.

The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them.

You only need a command to rm rf one box, if you are distributing that command to every box.

Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day.

The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.

bigbuppo 2 hours ago
And the most telling thing about most of these outages is that the provider later admits in their postmortem that they just didn't really understand how the system they made worked until it fell over and were forced to learn how it really works.

It's the sort of thing that used to keep me up at night.

Arainach 4 hours ago
The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.
bigbuppo 2 hours ago
Do you not remember crowdstrike?
Arainach 1 hour ago
Again: an outage caused by a config change is different from data loss.

The remediation was painful but it was not data loss.

bigbuppo 18 minutes ago
Yet.
ocdtrekkie 4 hours ago
I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low.

But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)

tokyobreakfast 6 hours ago
That seems unlikely. Is Google run by one Homer Simpson?
bigbuppo 6 hours ago
Yes.
JyB 5 hours ago
I don’t know if you’re being serious but that’s laughable
SchemaLoad 4 hours ago
The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them.

Though I'm sure the major players are all over this risk which is why it hasn't happened.

aragilar 41 minutes ago
Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.
jsheard 6 hours ago
There's at least five free ACME CAs, with failover it doesn't matter all that much if one of them falls over. If all of them fall over at once there's probably a more pressing issue like nuclear holocaust or alien invasion going on.
tokyobreakfast 6 hours ago
How many servers are set up with CA redundancy? I've yet to see one let alone hear of this practice.
jsheard 6 hours ago
For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any.

Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.

thayne 5 hours ago
They aren't all drop in replacements for each other though. For example, Let's Encrypt offers free wildcard certs (with dns verification), but for ZeroSSL, it requires a paid subscription.
jsheard 5 hours ago
ZeroSSL is weird, if you use their classic non-ACME interface then the free tier is indeed limited to 3 active certs which can't be wildcards, but if you use ACME then there's no limits and wildcards are allowed.

https://zerossl.com/documentation/acme/

> By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.

antonvs 4 hours ago
So the question is why this hit Youtube and Youtube TV so hard. Presumably they’re relying on ephemeral instances being able to get certs immediately, or something like that.

(Or an unrelated failure, of course)

msie 6 hours ago
I was thinking about the time some software influencer said that if you are afraid to deploy on Friday then there's something wrong with you. Eff that! Murphy's Law! (allen holub - https://x.com/allenholub/status/1637111242610610182)
tzs 5 hours ago
I often deployed on Friday evening. Several factors contributed to this decision.

1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers.

2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers.

3. I always made it so reverting would be easy.

4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.

3 hours ago
rconti 1 hour ago
> The fix has been rolled out and the issuance flow has been undrained. We again apologize for the inconvenience.

issuance flow has been undrained?

aaomidi 57 minutes ago
Draining is terminology they use for draining traffic from a service.
h4ch1 6 hours ago
Thought my Revanced patch got outdated for a second. Phew.
ddtaylor 6 hours ago
Have you had to update microG yet?
OhMeadhbh 6 hours ago
I worked at RSADSI when I was a kid and supported the custom spin of TIPEM Hayden and Sophia used at Verisign. This brings back some very bad memories.

But... hopefully... people created overlapping windows of cert validity so there's always a valid cert available for their services and can tolerate the CA being out of action for 8(?) hours. Imagine if your TGS/Kerberos or AWS IAM IdP was down for 8 hours.

antonvs 4 hours ago
For persistent services using the affected ACME API, the window is usually 30 days.

But that didn’t stop Youtube and Youtube TV from going down hard. I imagine they’re provisioning ephemeral VMs or service instances and relying on them being able to get certs immediately, or something like that.

dijit 6 hours ago
youtube (recommendations/homepage) also seems down, I wonder if its relater.
dyauspitr 6 hours ago
I can see all the videos and play the ones in my subscription tab though.
tokyobreakfast 6 hours ago
Oh no, whatever will we do without the inundation of e-thot shorts and AI-generated weight loss snake oil scam videos?
bethekidyouwant 6 hours ago
Never see these. Skill issue?
tokyobreakfast 6 hours ago
I'm inundated with them. YT has become borderline unusable. The homepage is nightmarish.

Can't search for anything without being overwhelmed with shorts in the results, many unrelated to what I'm searching.

bawolff 4 hours ago
YT pushing videos can be annoying, but at the same time there is an element of human free will here. You can chose not to watch them.
LPisGood 6 hours ago
I also never get these. It might be because you interact with them.
tokyobreakfast 6 hours ago
I browse logged out. Interact when them I do not. The weight loss and solar scams are forced advertisements before every video.
tfsh 6 hours ago
> I browse logged out. Interact when them I do not.

The logged out experience is closer to the interests of the average person. So if you're not pruning (and savings) your interests, that's hardly surprising.

tokyobreakfast 6 hours ago
The average person wants to be served AI slop and scams?
LPisGood 5 hours ago
What the average person says they want and what they will actually chose behaviorally will often not line up.
antonvs 5 hours ago
> I browse logged out.

This is like the guy who goes to the doctor complaining of eye pain whenever he drinks tea. "Have you tried taking the teaspoon out?"

pvab3 4 hours ago
Just block them. I haven't seen a short in months.
sciencesama 6 hours ago
Not sure but it is very strange i was served a strange tom And jerry video https://youtu.be/rilFfbm7j8k
nitinreddy88 6 hours ago
You can watch any YT video by directly following a link or from history/playlist etc. Its just their homepage etc is down
PLenz 6 hours ago
Eight hour estimated restoration time!
edwaldojunior 6 hours ago
Time to go over my Watch Later list
pkulak 6 hours ago
Here's a direct link to the latest Veritasium. You're welcome!

https://www.youtube.com/watch?v=cMx139eTxoc

6 hours ago
Thaxll 6 hours ago
Hmm why youtube does not work but google.com does.

Now I'm wondering if you rely on OCSP in a TLS client and the pki is Google does it still works?

arcfour 5 hours ago
OCSP is deprecated and basically dead at this point. Some clients still use it but I don't think many (any?) have actually enforced OCSP for years since it was notoriously fickle anyways.
kbelder 6 hours ago
Interesting. If you go to youtube.com it's all messed up; missing all the videos in the listings. But if you follow a video embedded in another site to youtube, it'll show and play fine. It'll break if you try to browse away from it.
arkryal2 6 hours ago
Yeah, YouTube is not one server, it's hundreds of them. The videos are served mostly from CDNs (the Content Distribution Network). It's a different set of servers than handles account logins, routing, etc.

Some Google Services are also down at the moment, unrelated to YouTube, so probably a failure along some common infrastructure pipeline.

Your History, Subscriptions and search should all work. You should be able to see any creator's page if you go to it directly. The videos are all still watchable. It's primarily the home page and recommended videos that are having issues. Basically any place they recommend videos you haven't seen is broken right now, but the videos are still there and accessible.

I've tried via VPN from the U.S., U.K., Sweden, Germany, Russia, Colombia, etc. Same issue across the board.

aaronmiler 6 hours ago
Heroku having service issues, dependency related?
flaxxer 6 hours ago
seeing heroku issues here too, had assumed it was salesforce's fault, bc of course they are eventually going to destroy heroku somehow, right?
spyrja 5 hours ago
Welp, looks like they're back up. Home page and notifications are loading just fine now.
rolph 6 hours ago
RobRivera 4 hours ago
Is that what was happening with my youtube mid workout?
arduanika 4 hours ago
Correct. It's not youtube, it's themtube.
Kapura 6 hours ago
Good thing I have nebula.tv for when youtube breaks
benatkin 6 hours ago
Isn't that the thing that a bunch of YouTube creators pitch inside their channels along with VPNs and supplements? I would never consider it because the ads rub me the wrong way. Or is it some alternative frontend for YouTube that happens to have a similar sounding name?
LPisGood 6 hours ago
It is a co-op where creators make videos without the threat of being demonetized or algorithmically punished - and it’s not garbage in the way you might expect people fearful of being demonetized might be.

Lots of excellent legal analysis, history, logistics, engineering content there.

It was initially founded by some of the most popular information YouTubers like CGPGrey, but he mysteriously left the project (I suspect one side wanted to be evil and the other side did not)

qmarchi 6 hours ago
Not quite. It's a co-op, where the creators own the shares of the company.

Supposedly a more holistic approach to video hosting with less oversight from the platform itself.

hylaride 6 hours ago
It's a place for creators to host long form content (that the google algorithm now disincentivizes) as well as history content that can't show a lot of history because of "violence" (like the holocaust).

Youtube is demonetizing channels left, right, and centre.

kittoes 6 hours ago
Nebula is actually quite a decent alternative/supplement to YouTube and worth the subscription IMHO.
1970-01-01 6 hours ago
Did someone buy the google.com domain again?
Shellban 2 hours ago
I have the domain. If you want you cat videos back, you are going to have to pay me:

ONE MILLION DOLLARS!

lawgimenez 6 hours ago
Down here in Southeast Asia
chiengineer 5 hours ago
While were all here does anyone want to launch a startup for a cloud security tool I built
manupati 6 hours ago
Still down
rvz 6 hours ago
Everyone loves to say they work at $FAMOUS_COMPANY, but when something like this happens, no-one will say that they did this.

Looking forward to the post-mortem.

wbsun 5 hours ago
Oh I am more than happy to tell people how I took down entire Google Cloud 11 years ago. I mean, of course to the level of details Google is comfortable with to share externally :)
LPisGood 6 hours ago
I mean, with any sufficiently large project or system it’s rarely super accurate to say one person did something.
microm 6 hours ago
All is down in eu too