You know when grown-ups tell you everything's going to be fine, and you think they're probably lying to make you feel better?
Everything's going to be fine.

Tuesday, May 05


Slow Samba


At my day job we have millions of images stored in a shared folder.  Each of around 800,000 objects is represented by about 20 images in different sizes and formats.

This is shared internally over Samba, because it works for what we need.

Except that for certain operations, that directory is horribly, painfully slow.  Local access on the file server is just fine; it's only remote access.

To cut a stupid story short, the problem is that Unix is case-sensitive and Samba is case-insensitive.  If you are looking for a file and Samba finds an exact match, it's as fast as you'd expect.  But if you are looking for a file and it's not there, Samba will scan the entire directory for case-insensitive matches, which is a disaster if you have millions and millions of files.


Create a specific share for that directory and set case sensitive = true.

Mount that in the appropriate place, and problem solved.

Posted by: Pixy Misa at 12:37 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 162 words, total size 1 kb.

Monday, May 04


Daily News Stuff 4 May 2020

Oops I Poked It Again Edition

Tech News

Anime Music Video of the Day

Disclaimer: In fact, scratch all of the ideas.

Posted by: Pixy Misa at 09:34 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 425 words, total size 4 kb.


Making An LXC Or LXD Container Accessible To The Internet

The simple manual approach:

Let's say your public IP range is  This is a pretty typical allocation for a dedicated server.

Your gateway will be, and your server will be  Broadcast is, so you have four IP addresses available.  

We'll pick, and we'll give the container the internal IP of  The simplest way to do that is to add the container names and internal IP addresses to your hosts file before launch.  The IP addresses will be assigned automatically and won't change.

The selected public IP should not be active on the host interface when you start.  If you ping it you should get no response.

  1. Add the public IP address within the container.

    root@container# ifconfig eth0:1 up

  2. Add a static route from the host server to the container.  (But see step 4 below.)

    root@host# route add gw

  3. If you try reaching the container now, chances are it won't work, because the route hasn't been announced to the gateway.  Fortunately, that's easy too.

    root@host# arp -Ds eth0 pub

    In this case eth0 will likely be something different.  You want the main public interface for your server - in my case this was enp35s0.

  4. Make sure to set up appropriate firewall rules in the container.  That static route will take effect before the filtering by ufw on the host.  (That one surprised me too.)

  5. That's it.  Done.

The problem with this approach is that it is manual and you'll need to make sure it's applied on reboot.  Just create an /etc/rc.local file on the host and on the container (on recent version of Ubuntu it doesn't exit by default), put the necessary commands in there, and end it with an exit 0.

The advantage of this approach is that since you're not mucking about with netplan files the chance of you locking yourself out of your own server is greatly reduced.

I wish someone had explained this clearly somewhere back when I had Mari, before I had to give it up because I couldn't get networking to work properly and couldn't afford to keep it around but not in production.  But Akane is a pretty nice server too.  On the CPU side it is much, much faster (about 2.5x single-threaded, about 2x multi-threaded), and it's less than half the price. 

Posted by: Pixy Misa at 05:28 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 399 words, total size 3 kb.


Invasion Of The Screen Scrapers

Someone found the author parameter and decided to try scraping every post on every blog on  From twenty different servers at once.  They've been blocked now.

Meanwhile, Akane running PyPy absolutely creams Aoi running Psyco.  PyPy was originally created by the same developer as Psyco but has a larger team and several years more active development, and it shows.  Combined with the faster CPU and the elimination of virtualisation overhead, the difference on this toy benchmark is on the order of 50x.

That's comparing our current JIT compiler to the new one, not interpreted to compiled.

Now I just have to update the template engine so that it doesn't trash that performance.  Accumulating the output using in-memory file operations should do the trick.

Update: Looks like PyPy is optimising some of the sub-tests into oblivion.  Adding some randomness into the benchmark to prevent that reduced it to only 6x faster.  Since the server also has twice the number of cores, that's 12x overall.

Posted by: Pixy Misa at 01:25 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 169 words, total size 1 kb.

Sunday, May 03


Daily News Stuff 3 May 2020

Your Sourcier Source For Newsier News Edition

Tech News

  • Akane (new server) is nearly three times faster than Aoi (old server) even when Aoi is cheating by using the Psyco JIT compiler.

    Psyco helps a lot in optimising loops and function calls - it's 20x faster on a benchmark that just loops and makes empty function calls - but can't speed up the Python code for functions like string manipulation.  Akane wins by close to 6x on string scanning.

    That's a lot more than the single-threaded difference between a Xeon E3-1230 and a Ryzen 3700X, so it's either a difference in cache sizes (8M vs. 32M) or virtualisation overheads.  Aoi is stuck running OpenVZ within KVM, where Akane is running LXD.

    I'll run the benchmark under PyPy, but I'll need to excise one of the tests.  I test large string concatenations because that's how Minx builds HTML.  That's very fast in Python (and Psyco) but offers nothing but pain and suffering under PyPy due to changes in the garbage collection.

  • I mention LXD and LXC a lot, which might be a bit confusing.  LXC is the container framework, and LXD is a management tool for LXC.  So when you are using LXD, the containers are LXC containers and you mostly use the lxc command to manage them.

    If you are using LXC directly you don't use the lxc command because that's part of LXD.  Instead you use commands like lxc-launch where LXD uses lxc launch.  And if you get the two mixed up you can scramble your configuration.



    I miss OpenVZ 6.

  • Also, the ASRock IPMI interface on their Ryzen server boards works.

    One of the long-standing problems with accessing IPMI over long distances (like Australia to the US) is keybounce on the KVM console.  I've often run into cases where it's all but unusable.  No sign of it here.  Ping times aren't wonderful, but it works regardless.

    This means that I can experiment safely with the network config on Akane to get LXD just the way I want it.

  • No we don't.  (Tech Crunch)

  • Nvidia's Hopper architecture may be headed for Samsung's 5nm process or it may not.  (WCCFTech)

    This is supposedly the next generation after Ampere, which is the next generation.

    Interesting point from the table is that the current top of the line Tesla V100 chip is larger than the estimated size of the rumoured 128 CU Navi part.  Whether AMD is working on such a chip is still unknown, but TSMC is capable of manufacturing it if they are.

  • A new California law requires that default passwords be unique for each unit manufactured.  (Serve the Home)

    The password is on the service tag.

    The service tag is on the server.

    The server is in a rack, in a datacenter, 10,000 miles away, and it's 3AM there.

    Which is not to say this law is a bad idea, for all that it came out of the Global Bad Law Research and Development Center in Sacramento.  But sysadmins will have occasion to curse, vehemently, in coming months.

  • There needs to be a Kickstarter for network documentation for LXD BECAUSE THERE FUCKING ISN'T ANY.

    There's a specific command, lxc network, to manage LXD networks, and there's no documentation for it at all, not even a man page.

    This is super painful when you are guessing your way to a solution because any time you screw up there's a good chance you'll lose access to your server.  Which is why I was testing IPMI today.

    Also, if you spin up an Ubuntu virtual machine rather than a container, it comes up with no working networking - and without the standard net-tools installed.  And you can't install net-tools because you have no networking to reach the install repo.  There is still the ip command but that is a pain in the butt.

Video of the Day

Speaking of benchmarks, Steve here speaks of problems of getting equal results over a wide range of systems, when individual CPUs could have anything from 2 to 64 cores.  He mentions one case that turned out to be nothing but a cache size test.  Irrespective of core counts and clock speeds, results were ordered by cache size.

Disclaimer: The one thing worse than network administration is network administration without an administration back-channel.  Though Google managed to break their own back-channel recently, so even that won't always save you.

Posted by: Pixy Misa at 09:52 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 741 words, total size 6 kb.

Saturday, May 02


Daily News Stuff 2 May 2020

Okay You Win Edition

Tech News

  • We've all had days like that.

  • And who hasn't done this?

  • Quibi is a failure while TikTok soars.  (TechDirt)

    This is because unlike Hollywood, China abandoned communism 40 years ago.

    True, they abandoned it for fascism.

  • Is Intel up to its old tricks again?

    Ian Cutress of AnandTech tweeted - and then deleted - that his sources were telling him Intel was using financial incentives to stop AMD-based NUCs.  Particularly now that AMD has a categorically superior chip for small form factor devices.

    This and many other questions are discussed, though not necessarily answered.

  • Currently running backups to the new server, after which I will figure out the details of the migration. can be switched from CentOS and OpenVZ to a current version of Ubuntu running in LXC behind a proxy with relatively little pain. 

    The main reason it's fiddly at all is that it uses Psyco - a precursor to PyPy - to speed up the code, but the new server is so much faster that we can do without for a while, until I can check that the code runs cleanly in PyPy,

    Well, also that MySQL does stupid stuff every time we do a migration, and loses key settings that break things but only after I've tested it all and gone to bed.

    The other stuff still runs on CPanel and not only requires CentOS but proper routing, which I have yet to get working cleanly with LXC.

    The big advantage of this move - apart from the dramatically better hardware - is ZFS, which lets me instantly snapshot and back up the virtual servers.  That will make future migrations much, much easier if we're willing to take a little downtime, because those snapshots can be restored on any other LXC host with everything intact.

    I used to do this with OpenVZ, but OpenVZ 6 has been deprecated and OpenVZ 7 is not compatible.

    Update: This is why I like LXC/LXD and OpenVZ 6.

    I can tweak the memory and CPU capacities of the virtual machines live.  Are we getting extra load on one app and need to give it another couple of cores and an extra 4GB of RAM for a while?  Click click, done.  No waiting, and most important, no reboots.

Anime Music Video of the Day

I hadn't seen this before, even though it's by Copycat Revolver who has done some of my favourite AMVs ever.

It does make me want to watch Sayonara Zetsubou Sensei.

Other Copycat Revolver AMVs you may have seen here before:

Disclaimer: So, that anime stuff...  What's good right now?  I have a 1.5TB MySQL database to migrate at my day job (to our new self-hosted cloud thingy) and could do with the distraction.

Posted by: Pixy Misa at 11:26 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 468 words, total size 4 kb.


Welcome Akane Mk III!

Naming this one Akane because (a) host nodes are named after anime girls whose names are also colours and (b) virtual machines are named after anime girls from the same series as the host node, and Ranma has a ton of characters so we won't run out.

Server specs:

Ryzen 3700X
100TB bandwidth over 1Gb
Ubuntu 20.04

I took a look at the SSD and it's a Samsung PM1725.  (Storage Review)

It looks like this is a 48-layer TLC model, but it's an enterprise drive rated for 5 DWPD - which equals 45,000 4k random writes per second for five years.  Since our current server averages, um, eight writes per second (though they're closer in size to 40k), I don't think we're going to have a problem there.

Posted by: Pixy Misa at 12:39 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 138 words, total size 1 kb.


Daily News Stuff 1 May 2020

3.2TB Of What Exactly Edition

Tech News

  • So at my day job we're now basically our own cloud platform.  Saves us about 85% in costs, except for bandwidth, where it saves us about 98%.  Bandwidth charges at AWS, Google, and IBM are absurd.

    Except for IBM's global private network, which is free.  I have to admit, that is brilliant if you have servers on multiple continents.

  • Intel has some new chips out.  (AnandTech)

    Yes, it's the 10th generation Comet Lake S range - or as the article points out, it's the 5th generation of Skylake, because the architecture hasn't changed for years.

    Although it's a boring launch - partly because all the leaks were 100% accurate, and partly because the fastest CPU in the lineup already loses to AMD's second fastest mainstream processor - there are at least a lot of product codes, ranging from the $42 35W dual core Celeron G5900T to the $488 125W (yeah, right) 10 core i9-10900K.

    Comparing Intel with Intel, that 10 core part represents about a 70% price cut over the past three years.  But at my day job we're ordering nothing but AMD servers now, and that trend is just going to grow.

  • A company has patented a general approach for banning social media posts that "spread misinformation".  (TechDirt)

    Which is great because now they can sue Twitter, Facebook, and YouTube into the ground.

    And the EU as well.

    Which is - wonder of wonders - actually standing up to China over the latter's propaganda efforts.

  • .Org is off.  (ICANN)

    The planned sale of the .org domain for $1.1 billion has been killed after attracting the attention of everyone from ICANN board members to state Attorneys General.

  • Is Xiaomi the new Huawei?  (Forbes)

    Well, for a start, don't use the Xiaomi browser.  How much this affects the phone itself with a non-spyware browser isn't clear.

  • Is there a server monitoring dashboard out there that doesn't look like a dog park after an outbreak of canine dysentery?

    Thinking of rolling my own using the SmartAdmin template.  It's not open source, but on the other hand it doesn't look like doggy diarrhea.  I already wrote the monitoring agent - that was my test project when learning Crystal.  The collector can be a simple Python + MySQL app.

  • Font Awesome's Duotone icons are pretty cute.  (FontAwesome)

    These came out last year so I'm a bit behind.  I was an early bird backer of their Kickstarter so I have not one but two perpetual Font Awesome Pro licenses.  So you can expect to see these icons show up here at some point.

  • Never let a good crisis go to waste.  (Substack)

    And if you can't find a crisis, you can always create one.  Matt Taibbi weighs in against the neo-fascists over at The Atlantic.

  • Mail in a Box is mail in a box.  (Mail in a Box)

    Basically it's a one-click mail server suitable for deploying on a $5 server at Digital Ocean, Vultr, or Amazon Lightsail.

    It's not very configurable, but if you've looked inside the workings of something like Modoboa, that is a good thing.  Email servers these days are a complete fucking nightmare.  Every idea that everyone has had about email for the past fifty years is still in there, and the config files are infinitely recursive.

  • Six lessons learned from a dead iMac.  (Tidbits)

    1. Target device mode is brilliant - you can boot one device off the disk drive in another over Thunderbolt.

    2 through 6. If it was a PC you could just pop it open and swap the drive without going through all this agony.  If swapping the disk drive in your computer makes the screen fall off - and you're thinking of buying another one just the same - then you have a problem.

  • New server has been ordered.  Ryzen 3700X, 64GB ECC RAM, 3.2TB Samsung NVMe SSD, 100TB bandwidth, $129 per month.  Will be running Ubuntu 20.04 unless I hit a problem during installation in which case I'll fall back to 18.04.

Disclaimer: No.

Posted by: Pixy Misa at 12:37 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 682 words, total size 6 kb.

Friday, May 01


New Server Maybe Incoming

Looking at a couple of options.  Not going straight for the 3960X given the terrible exchange rates at the moment, but a 3700X with 128GB 64GB of RAM and 5TB 3.2TB of NVMe SSD is possible.

Update: They ran out of the drives that would have given me the 5TB configuration, and 128GB is a bit expensive if I want ECC (which I do).  This configuration is still faster, has more SSD, and nearly as much RAM as all three current servers combined.  Oh, and costs much, much less.

I still need to figure out how to get the networking right for CPanel under LXC.  I think I know, now.

It doesn't need to be my dream config because they have similar pricing on different configs every week.  The reason they're cheap is the same reason the config I wanted isn't available: They buy bulk lots of new-in-box but superseded enterprise components at about 25% of retail price, and when they're gone, they're gone.

At my day job we just deployed 48TB of MLC SSD with the same hosting company for surprisingly little cost - but that drive model launched at the end of 2016.

Posted by: Pixy Misa at 11:57 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 199 words, total size 1 kb.

<< Page 4 of 4 >>
80kb generated in CPU 0.0214, elapsed 0.4074 seconds.
51 queries taking 0.3914 seconds, 359 records returned.
Powered by Minx 1.1.6c-pink.
Using http / / 357