I'm in the future. Like hundreds of years in the future. I've been dead for centuries.
Oh, lovely, you're a cheery one aren't you?
Oh, lovely, you're a cheery one aren't you?
Wednesday, August 04
Server Problems
This server is crashing almost every day right now.
I have a new server, and I have complete and up-to-date backups on the new server.
What I haven't had so far is any time to configure this system on the new server.
Should happen in the next few days. I pulled about a forty-hour week just between 5PM Friday and 9AM Monday, but that was the last drama from that product launch, and the next launch isn't for, oh, a week at least.
It's not until October that things will get really crazy.
I did get a raise, and we've hired a bunch of new staff, so it should get less crazy by the end of the year. I just need to survive long enough to see it...
This server is crashing almost every day right now.
I have a new server, and I have complete and up-to-date backups on the new server.
What I haven't had so far is any time to configure this system on the new server.
Should happen in the next few days. I pulled about a forty-hour week just between 5PM Friday and 9AM Monday, but that was the last drama from that product launch, and the next launch isn't for, oh, a week at least.
It's not until October that things will get really crazy.
I did get a raise, and we've hired a bunch of new staff, so it should get less crazy by the end of the year. I just need to survive long enough to see it...
Posted by: Pixy Misa at
12:20 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 130 words, total size 1 kb.
Thursday, May 20
Google In The Apiary With A Lead Pipe
The server was getting overloaded with crappy requests again, but I couldn't see any difference between the crappy requests overloading the server and the usual crappy requests that only take about 50 milliseconds and cause no problems at all.
Except that we were also getting indexed by Google and the Google bot was tracking links to RSS feeds in places where RSS feeds don't really belong but the server will do its best to fulfil anyway.
So I blocked a couple of those. Not all of them, just a couple.
And the problem was resolved.
The server was getting overloaded with crappy requests again, but I couldn't see any difference between the crappy requests overloading the server and the usual crappy requests that only take about 50 milliseconds and cause no problems at all.
Except that we were also getting indexed by Google and the Google bot was tracking links to RSS feeds in places where RSS feeds don't really belong but the server will do its best to fulfil anyway.
So I blocked a couple of those. Not all of them, just a couple.
And the problem was resolved.
Posted by: Pixy Misa at
03:51 PM
| Comments (5)
| Add Comment
| Trackbacks (Suck)
Post contains 103 words, total size 1 kb.
Sunday, April 18
The Prodigal Server Returns
Yes, Akane is back - our shiny Ryzen 3700X, with its 64GB of ECC RAM and enterprise NVMe storage - in a shade under two weeks.
Doing a full offsite backup, followed by software updates, then we can return to something approaching normality.
Meanwhile I'm getting errors on the backup drive I had them swap into the new server at my day job, which is annoying because I'll need to re-take or re-verify 11TB of backups, but nothing is actually down which is refreshing.
Yes, Akane is back - our shiny Ryzen 3700X, with its 64GB of ECC RAM and enterprise NVMe storage - in a shade under two weeks.
Doing a full offsite backup, followed by software updates, then we can return to something approaching normality.
Meanwhile I'm getting errors on the backup drive I had them swap into the new server at my day job, which is annoying because I'll need to re-take or re-verify 11TB of backups, but nothing is actually down which is refreshing.
Update: Backups of our server are complete. That server is so fricking fast compared to this one. I'd like to replace it with a 5900X but it's already really fast.
Backups of the backups of the work server are ongoing, since there's 60x as much data over there. (11TB vs. 180GB.)
Backups of the backups of the work server are ongoing, since there's 60x as much data over there. (11TB vs. 180GB.)
Oh, and the HoloEN Minecraft server is back too.
Update: One of the things that worried me about the main server being down for so long was that I didn't have an up-to-date offsite backup. I did have offsite backups, just nothing less than a month old.
That server is running LXD virtualisation on ZFS. This gives you two ways to do backups:
That server is running LXD virtualisation on ZFS. This gives you two ways to do backups:
lxc snapshotwhich is simple and instantaneous and uses minimal disk space but is stored on the local drivelxc exportwhich gives you a complete portable backup in a single file but by default backs up your everything straight into your root filesystem
So if your container is larger than the available space in your root filesystem, splat.
You can configure it not to do that, but it's not very well documented, and by not very well documented I mean have fun trawling through Stack Overflow, sucker.
Anyway, since right now everyone is on this server and that server is free, I thought I'd try updating the software and configuring proper backups with
Tried it on a small container - around 1GB - with pigz (parallelised GZip) compression, and it completed in 15 seconds. Great!
You can configure it not to do that, but it's not very well documented, and by not very well documented I mean have fun trawling through Stack Overflow, sucker.
Anyway, since right now everyone is on this server and that server is free, I thought I'd try updating the software and configuring proper backups with
lxc export.Tried it on a small container - around 1GB - with pigz (parallelised GZip) compression, and it completed in 15 seconds. Great!
Tried it on the main blog container, which is around 70GB of active data, and so far it's been running for two and a half hours, has used 470GB of temporary space, and hasn't even started writing the backup file yet.
Oh, and it doesn't give you any progress information, not even Microsoft level where the indicator sometimes runs backwards.
And you can't stop a running backup.
This is garbage.
Oh, and it doesn't give you any progress information, not even Microsoft level where the indicator sometimes runs backwards.
And you can't stop a running backup.
This is garbage.
Update: Always use --instance-only to avoid the snapshot explosion, and to cancel a running export,
kill the pigz; the backup process will abort cleanly. Meanwhile I'm watching Kanata building and tuning an array of Minecraft note blocks and redstone delay thingies to play Hololive songs by ear. She doesn't even count, she just goes wapwapwapwapwap on one block after another and it comes out right. Gen 4's own little Beethoven.
Posted by: Pixy Misa at
01:09 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 480 words, total size 4 kb.
Monday, April 12
Captain's Log, Star Date Whatever The Hell Today Is
Deployed an Nginx instance configured as a caching proxy and it seems to be helping out a lot. Load average has dropped from 40 to - right now - 2. Wait, 10. Wait, 7. It's still bouncing around a but but not getting out of control as it was earlier.
That's a combination of (1) disabling sessions on static files, (2) caching said static files, and (3) people not impatiently hitting F5 when the site is slow to load because the site mostly isn't slow to load.
I didn't much enjoy this bit, though:
Deployed an Nginx instance configured as a caching proxy and it seems to be helping out a lot. Load average has dropped from 40 to - right now - 2. Wait, 10. Wait, 7. It's still bouncing around a but but not getting out of control as it was earlier.
That's a combination of (1) disabling sessions on static files, (2) caching said static files, and (3) people not impatiently hitting F5 when the site is slow to load because the site mostly isn't slow to load.
I didn't much enjoy this bit, though:
2021/04/12 13:29:43 [emerg] 4954#4954: "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /etc/nginx/nginx.conf:66
All I can say is that the trains would collide head-on just outside Albany.
Posted by: Pixy Misa at
11:47 PM
| Comments (1)
| Add Comment
| Trackbacks (Suck)
Post contains 135 words, total size 1 kb.
Monday, April 05
Older Posts Temporarily Hidden
Power is out at the new hosting company where Ace lives - and may be out for a day or two. After power went out, they switched to generator backup.
Then one of the backup generators caught fire.
Power is out at the new hosting company where Ace lives - and may be out for a day or two. After power went out, they switched to generator backup.
Then one of the backup generators caught fire.
It was promptly put out, but power has been cut to the entire building and won't be restored until there's been a safety inspection.
I had a whole bunch of his content already in this server and activated his site over here, but that slowed the server to a crawl, and I had to hide older posts while I sorted that out.
All the content is safe and will be progressively re-enabled. Should all be back tomorrow.
I had a whole bunch of his content already in this server and activated his site over here, but that slowed the server to a crawl, and I had to hide older posts while I sorted that out.
All the content is safe and will be progressively re-enabled. Should all be back tomorrow.
Update: We also have four servers at the same datacenter at my day job. It's the big data crunching cluster I mentioned before, including the massive 128-core Epyc server.
That server - with complete backups of all the critical data - is now back online. So is the one that caused me so much trouble that my boss authorised a 128-core replacement without a second thought.
I was half expecting none of the servers to survive except the broken one. At least I'm spared that nightmare.
That server - with complete backups of all the critical data - is now back online. So is the one that caused me so much trouble that my boss authorised a 128-core replacement without a second thought.
I was half expecting none of the servers to survive except the broken one. At least I'm spared that nightmare.
Posted by: Pixy Misa at
02:24 PM
| Comments (3)
| Add Comment
| Trackbacks (Suck)
Post contains 205 words, total size 1 kb.
Monday, March 08
Maintainenances
Was trying to purge the sessions table (which was getting rather large) and the database got unhappy and the site (or rather, sites, since there's somehow over 100,000 of them) slowed to a crawl.
Instead I copied the entries I wanted to keep into a new table and swapped tables.
Happier now.
Was trying to purge the sessions table (which was getting rather large) and the database got unhappy and the site (or rather, sites, since there's somehow over 100,000 of them) slowed to a crawl.
Instead I copied the entries I wanted to keep into a new table and swapped tables.
Happier now.
Posted by: Pixy Misa at
11:50 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 53 words, total size 1 kb.
Sunday, January 10
Parler Embed Test
Ah, got it. It works just fine, but glitches slightly if you embed the same post twice in the page. That's not really a problem; I'll let them know and they can fix that when they have slightly fewer things to deal with.
Update: Implemented for testing here.
Update: Implemented for testing here.
Posted by: Pixy Misa at
02:26 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 51 words, total size 1 kb.
Saturday, January 09
Video Spam Test Post
If this works properly, this post won't break your browser.
Update: But your browser might break it.
Waterfox Classic's DOM support is out of date - deliberately, so as not to break plugins - and Pale Moon's JavaScript support is way out of date.
I'll need to detect them via the user agent string and revert to direct (slow) embedding.
If this works properly, this post won't break your browser.
Update: But your browser might break it.
Compatibility Test Results
Desktop- Chrome ✔
- Firefox ✔
- Edge ✔
- Vivaldi ✔
- Brave ✔
- Pale Moon âŒ
- Opera - not tested because I uninstalled it a year ago
- Safari ✔
- Waterfox 3G ✔
- Waterfox Classic âŒ
Waterfox Classic's DOM support is out of date - deliberately, so as not to break plugins - and Pale Moon's JavaScript support is way out of date.
I'll need to detect them via the user agent string and revert to direct (slow) embedding.
Huh. It works.
Once more, withfeeling playlists.
Once more, with
That was... Surprisingly painless.
Posted by: Pixy Misa at
12:39 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 111 words, total size 2 kb.
Sunday, January 12
Feeding The Dog
Oops. Comments should be fixed now.
The bug only affected this site, which was exactly the point of splitting it off to its own environment.
Oops. Comments should be fixed now.
The bug only affected this site, which was exactly the point of splitting it off to its own environment.
Posted by: Pixy Misa at
02:45 PM
| Comments (1)
| Add Comment
| Trackbacks (Suck)
Post contains 28 words, total size 1 kb.
Tuesday, October 29
Site Hic-Hic-Hic-Hiccups
Not my fault this time!
Not my fault this time!
Posted by: Pixy Misa at
11:13 AM
| Comments (2)
| Add Comment
| Trackbacks (Suck)
Post contains 7 words, total size 1 kb.
59kb generated in CPU 0.0242, elapsed 0.1308 seconds.
54 queries taking 0.1144 seconds, 216 records returned.
Powered by Minx 1.1.6c-pink.
54 queries taking 0.1144 seconds, 216 records returned.
Powered by Minx 1.1.6c-pink.
