Sunday, December 29
Daily News Stuff 29 December 2019
Only Two More Shopping Days Until New Year's Edition
Only Two More Shopping Days Until New Year's Edition
Tech News
- The Radeon 5600XT looks to be a Radeon 5700 (non-XT) with the memory bus cut from 256 bits to 192 unless it isn't. (WCCFTech)
ASRock leaked the details of an upcoming card, confirming clocks, shader counts, memory speeds and size, everything but the TDP and price, basically.
- FreeDB is shutting down. (Hacker News)
FreeDB was forked from CDDB when that went project commercial. It was acquired by Magic in 2006, so they've been maintaining it for free for 13 years now.
- Wyze, a vendor of IoT-enabled cameras and door locks, suffered a data breach that exposed the details of 2.4 million customers. (ZDNet)
The culprit was - you guessed it - Elasticsearch.
- Google Stadia will add yet another Tomb Raider game in January. (WCCFTech)
At that point, 20% of all games available on Stadia will be Tomb Raider.
- LG will be showing off an automated vegetable garden at CES. (The Verge)
It is the 21st century. I guess.
- I was going over some benchmark scores today, and the Ryzen 3900X offers a bigger performance boost over my current 1700 than the 1700 does over the old FX-8150 Bulldozer chips. On single-threaded tasks, never mind multi-threaded.
I'm still going to hold out for Ryzen 3.
- YouTube has been evil for a long time. (Archive.org)
Back in 2008 they were in discussions with the developer of a small site built using the YouTube APIs, and offered to showcase him as a featured developer.
At exactly the same time they changed their TOS to make sure his site could never earn a penny.
Don't ask why the scorpion stings you.
New System Notes
Doing API testing on the new system today. Request routing, logins, sessions, cookies, automatic compression, all that good stuff.
The query that I was worried would slow down as the database grows - the standard user timeline - does indeed slow down as the database grows. I built in an engine to take care of that, and today I wrote the necessary query to use that engine.
Since that new query currently takes the time to build a timeline from 0.03s to 0.00s I'm adding more data to my test system to measure it again.
Oh, and you can search within your timeline. Twitter lets you do that now too, though.
Update: Stack engine vs. timeline engine:
With a small database the timeline query was running fine. But if the system had taken off it would have been Fail Whale Squared. (I think this type of query caused about 90% of Twitter's problems in the early days.)
Stack requests automatically remember the last N items in your timeline so they don't have to mess around finding them again.
The other major mode is the channel request, which are used for blogs and forums and things like that. Those have no problems:
That's the API request time, by the way, not just the database request, though for the timeline the database request is the overwhelming majority of the time.
I knew about this before but hadn't done the optimisation, because having a standard query let me enforce privacy checks in a single central location. Now I have three versions of the query and have to make sure the privacy checks are applied to each one.
Now I'm wondering if I can fix up that timeline query to make it run faster, because that could be really useful...
Update: Hah! Yes, that works. If I need to rebuild a user's stack, I can find the IDs of the last thousand posts that should appear in their timeline and shove them into their stack in 60 milliseconds flat. Then database queries within the stack take about 4ms.
The idea there is that if you don't log in for a while the system will stop updating your stack to save resources, but when you do log in I want it brought up to date quickly enough that you don't really notice it. 60ms is fine.
The main message query has five joins and ten subselects, which is great when the optimisation is just right because it gets everything the API needs in one go. When the optimisation is not just right, though, things go south in a hurry.
The stack works great because it means the main query never has to sort - to get the top 20 messages it just reads the top 20 stack records in index order and does five one-to-one joins.
The query that I was worried would slow down as the database grows - the standard user timeline - does indeed slow down as the database grows. I built in an engine to take care of that, and today I wrote the necessary query to use that engine.
Since that new query currently takes the time to build a timeline from 0.03s to 0.00s I'm adding more data to my test system to measure it again.
Oh, and you can search within your timeline. Twitter lets you do that now too, though.
Update: Stack engine vs. timeline engine:
500 timeline requests in 69.248s, avg request 139.1ms
500 stack requests in 3.963s, avg request 7.9ms
With a small database the timeline query was running fine. But if the system had taken off it would have been Fail Whale Squared. (I think this type of query caused about 90% of Twitter's problems in the early days.)
Stack requests automatically remember the last N items in your timeline so they don't have to mess around finding them again.
The other major mode is the channel request, which are used for blogs and forums and things like that. Those have no problems:
500 channel requests in 3.042s, avg request 6.1ms
That's the API request time, by the way, not just the database request, though for the timeline the database request is the overwhelming majority of the time.
I knew about this before but hadn't done the optimisation, because having a standard query let me enforce privacy checks in a single central location. Now I have three versions of the query and have to make sure the privacy checks are applied to each one.
Now I'm wondering if I can fix up that timeline query to make it run faster, because that could be really useful...
Update: Hah! Yes, that works. If I need to rebuild a user's stack, I can find the IDs of the last thousand posts that should appear in their timeline and shove them into their stack in 60 milliseconds flat. Then database queries within the stack take about 4ms.
The idea there is that if you don't log in for a while the system will stop updating your stack to save resources, but when you do log in I want it brought up to date quickly enough that you don't really notice it. 60ms is fine.
The main message query has five joins and ten subselects, which is great when the optimisation is just right because it gets everything the API needs in one go. When the optimisation is not just right, though, things go south in a hurry.
The stack works great because it means the main query never has to sort - to get the top 20 messages it just reads the top 20 stack records in index order and does five one-to-one joins.
Disclaimer: I tried to recite "How Doth the Little Busy Bee" but it came out all different.
Posted by: Pixy Misa at
10:47 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 799 words, total size 6 kb.
52kb generated in CPU 0.0119, elapsed 0.1079 seconds.
56 queries taking 0.1 seconds, 347 records returned.
Powered by Minx 1.1.6c-pink.
56 queries taking 0.1 seconds, 347 records returned.
Powered by Minx 1.1.6c-pink.