It was a bad day. A lot of bad stuff happened. And I'd love to forget it all. But I don't. Not ever. Because this is what I do. Every time, every day, every second, this: On five, we're bringing down the government.

Sunday, December 04

Blog

We're Back

Well, that was exciting.

We've moved to the new server.  Well, two new servers, since the designated new server had weird issues.

The migration was complicated by the old server disappearing for 24 hours just before I was set to start doing this.

But...  We're back.

We were actually back for a while earlier (as some people noted) but that was with three weeks of data missing.  In the end I was able to recover everything up to the Scheduled Maintenance Window of Doom, and the only person who posted anything after that was me (and a couple of commenters) so that's close enough to everything.

Also, looks like I can stop paying for that backup server, since it's kind of deceased.

Posted by: Pixy Misa at 01:58 PM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 124 words, total size 1 kb.

Wednesday, September 28

Blog

Poop

Server took an unscheduled nap.

Again.

Posted by: Pixy Misa at 02:00 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 7 words, total size 1 kb.

Saturday, September 03

Blog

Well, That Just Fills Me With Confidence

We recive complain. If not be resolved after 24H your services will be closed
As if I needed another reason to get off this server.

Looks like the hosting has quietly changed hands again.  Yes, there are some crappy sites here on mee.nu - offering free blog hosting will do that - and I need to root them out.  I'll likely need to disable signups for a while.

Everything is backed up because I don't trust them at all not to simply switch off the servers.  This server co-operated by crashing during the backup.

Posted by: Pixy Misa at 04:32 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 100 words, total size 1 kb.

Sunday, April 24

Blog

Restoree

Finally have a script running to finish restoring all the posts from the backup into the live database after the Big Mess last year when the other datacenter caught fire and everyone had to squeeze onto a single $50 server for three weeks.

It's moving pretty quickly so if your posts from 2012-2016 aren't back yet they should be within the next few hours.

Currently reading Mahoutsukai no Insatsujo / A Witch's Printing Office, in which the protagonist establishes Magiket.
 

Posted by: Pixy Misa at 10:17 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 81 words, total size 1 kb.

Monday, January 17

Blog

Well, Poop

The site has been fixed.

Redis failed to start after a server problem, and then the virtual server filled up with email alert attempts that weren't going out for some reason.

I have to get us moved.

Posted by: Pixy Misa at 08:39 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 39 words, total size 1 kb.

Saturday, September 25

Blog

Content Not Relevant, Testing Something

I mean, not relevant to this particular post, it is relevant o the world at large and the frauds in charge.



Also, this is literally the objectification of women.

Posted by: Pixy Misa at 06:14 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 235 words, total size 3 kb.

Wednesday, August 04

Blog

Server Problems

This server is crashing almost every day right now.

I have a new server, and I have complete and up-to-date backups on the new server.

What I haven't had so far is any time to configure this system on the new server.

Should happen in the next few days.  I pulled about a forty-hour week just between 5PM Friday and 9AM Monday, but that was the last drama from that product launch, and the next launch isn't for, oh, a week at least.

It's not until October that things will get really crazy.

I did get a raise, and we've hired a bunch of new staff, so it should get less crazy by the end of the year.  I just need to survive long enough to see it...

Posted by: Pixy Misa at 12:20 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 130 words, total size 1 kb.

Thursday, May 20

Blog

Google In The Apiary With A Lead Pipe

The server was getting overloaded with crappy requests again, but I couldn't see any difference between the crappy requests overloading the server and the usual crappy requests that only take about 50 milliseconds and cause no problems at all.

Except that we were also getting indexed by Google and the Google bot was tracking links to RSS feeds in places where RSS feeds don't really belong but the server will do its best to fulfil anyway.

So I blocked a couple of those.  Not all of them, just a couple.

And the problem was resolved.

Posted by: Pixy Misa at 03:51 PM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 103 words, total size 1 kb.

Sunday, April 18

Blog

The Prodigal Server Returns

Yes, Akane is back - our shiny Ryzen 3700X, with its 64GB of ECC RAM and enterprise NVMe storage - in a shade under two weeks.

Doing a full offsite backup, followed by software updates, then we can return to something approaching normality.



Meanwhile I'm getting errors on the backup drive I had them swap into the new server at my day job, which is annoying because I'll need to re-take or re-verify 11TB of backups, but nothing is actually down which is refreshing.

Update: Backups of our server are complete.  That server is so fricking fast compared to this one.  I'd like to replace it with a 5900X but it's already really fast.

Backups of the backups of the work server are ongoing, since there's 60x as much data over there.  (11TB vs. 180GB.)

Oh, and the HoloEN Minecraft server is back too.


Update: One of the things that worried me about the main server being down for so long was that I didn't have an up-to-date offsite backup.  I did have offsite backups, just nothing less than a month old.

That server is running LXD virtualisation on ZFS.  This gives you two ways to do backups:
  1. lxc snapshot which is simple and instantaneous and uses minimal disk space but is stored on the local drive
  2. lxc export which gives you a complete portable backup in a single file but by default backs up your everything straight into your root filesystem

So if your container is larger than the available space in your root filesystem, splat.

You can configure it not to do that, but it's not very well documented, and by not very well documented I mean have fun trawling through Stack Overflow, sucker.

Anyway, since right now everyone is on this server and that server is free, I thought I'd try updating the software and configuring proper backups with lxc export.

Tried it on a small container - around 1GB - with pigz (parallelised GZip) compression, and it completed in 15 seconds.  Great!

Tried it on the main blog container, which is around 70GB of active data, and so far it's been running for two and a half hours, has used 470GB of temporary space, and hasn't even started writing the backup file yet.

Oh, and it doesn't give you any progress information, not even Microsoft level where the indicator sometimes runs backwards.

And you can't stop a running backup.

This is garbage.

Update: Always use --instance-only to avoid the snapshot explosion, and to cancel a running export, kill the pigz; the backup process will abort cleanly.  

Meanwhile I'm watching Kanata building and tuning an array of Minecraft note blocks and redstone delay thingies to play Hololive songs by ear.  She doesn't even count, she just goes wapwapwapwapwap on one block after another and it comes out right.  Gen 4's own little Beethoven.

Posted by: Pixy Misa at 01:09 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 480 words, total size 4 kb.

Monday, April 12

Blog

Captain's Log, Star Date Whatever The Hell Today Is

Deployed an Nginx instance configured as a caching proxy and it seems to be helping out a lot.  Load average has dropped from 40 to - right now - 2.  Wait, 10.  Wait, 7.  It's still bouncing around a but but not getting out of control as it was earlier.

That's a combination of (1) disabling sessions on static files, (2) caching said static files, and (3) people not impatiently hitting F5 when the site is slow to load because the site mostly isn't slow to load.

I didn't much enjoy this bit, though:

2021/04/12 13:29:43 [emerg] 4954#4954: "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /etc/nginx/nginx.conf:66

All I can say is that the trains would collide head-on just outside Albany.

Posted by: Pixy Misa at 11:47 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 135 words, total size 1 kb.

<< Page 1 of 43 >>
59kb generated in CPU 0.016, elapsed 0.103 seconds.
54 queries taking 0.0914 seconds, 216 records returned.
Powered by Minx 1.1.6c-pink.
Using https / https://ai.mee.nu / 214