I have spent hours arguing with someone at my work that the issue we are experiencing at our remote locations is not due to bandwidth, but latency. These graphics are exactly what I've been looking for to help get my point across.
People do a speedtest and see low (sub-100) numbers and think that's why their video call is failing. Never mind the fact that Zoom only needs 3 Mbps for 1080p video.
Latency is a cruel mistress. Had a Customer who was using an old Win32 app that did a ton of individual SELECT queries against the database server to render the UI. They tried to put it on the end of a VPN connection and it was excruciating. The old vendor kept trying to convince them to add bandwidth. Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.)
speedtest.net added support for tracking latency under load a few years ago. they show ping during up/dl now. That's the number to show your colleague.
However they tend to use something like the 75th percentile and throw out real data. The waveform bufferbloat test does 95% and supplies whisker charts. cloudflare also.
No web test tests up + down at the same time, which is the worst scenario. crusader and flent.org's rrul test do.
Rathan than argue with your colleague, why not just slap an OpenWrt box as a transparent bridge inline and configure CAKE SQM?
Would you put the "OpenWrt box as a transparent bridge inline" between your home router and the cable modem, or on the house side of the home router?
I would replace the home router with an OpenWrt router.
These type of solutions don’t scale to large ISPs, and gets costly to deploy at the edge. It’s also not just about throughput in Gbps, but Mpps.
Also, this doesn’t take into account that the congestion/queueing issue might be at an upstream. I could have 100g from the customers local co to my core routers, but if the route is going over a 20g link to a local IX that’s saturated it probably won’t help to have fq/codel at the edge to the customer.
I know it's common to say bandwidth casually, but I really wish a Blog trying to explain the difference between data rate and latency would not conflate bandwidth and data rate (one could also say throughput or capacity although the latter is also technically incorrect). The term bandwidth really denotes the spectral width occupied by a signal, and while it is related to the actual data rate, it is much less so nowadays where we use advanced modulation compared to back when everything was just OOK.
Coincidentally, the difference between latency and data rate is also much clearer using these two terms.
Working from home has really put a spotlight on the terrible asymmetric upload speeds of most cable internet.
I can get 1 gb down but only 50 mb upload. Certain tasks (like uploading a docker image) I cant do at all from my personal computer.
The layman has no idea the difference, and even most legislators don't understand the issue ("isn't 1 gb fast enough?")
This. I've been fighting AT&T for awhile because they told the FCC (via their broadband maps [0]) that they supply fiber to my condo, so I bought it expecting to get fiber. Well when I finally go to set up internet service, they only offer 50/5 DSL service. Fortunately I can get cable that has usable down speeds but the up is substantially less than 50 with garbage routing.
I'm not very happy.
I got lucky and fiber became available in my neighborhood around the same time I noticed how painful pushing images over cable was. Hopefully you'll get that option soon too.
For the unlucky, maybe we can take advantage of the fact that most image pushes have a predecessor which they are 99% similar to. With some care about the image contents (nar instead of tar, gzip --rsyncable, etc) we ought to be able to save a lot of bandwidth by using rsync on top of the previous version instead of transferring each image independently.
> I can get 1 gb down but only 50 mb upload. Certain tasks (like uploading a docker image) I cant do at all from my personal computer.
As someone who used to work with LLMs, I feel this pain. It would take days for me to upload models. Other community members rent GPU servers to do the training on just so that their data will already be in the cloud, but that's not really a sustainable solution for me since I like tinkering at home.
I have around the same speeds, btw. 1Gb down and barely 40Mb up. Factor of 25!
I feel your pain, I haven't been in ML world directly for a few years now but I've done the same exercise multiple times.
The worst part is that block compression actually does not help if it doesn't do a significantly good job of compression AND decompression. My use case had to immediately deploy the models across a few nodes in a live environment at customer sites. Cloud wasn't an option for us and fiber was also unavailable many times.
The fastest transport protocol was someone's car and a workday of wages.
> Now a company with bad performance can ask its ISP to fix it and point at the software and people who have already used it. If the ISP already knows it has a performance complaint, it can get ahead of the problem by proactively implementing LibreQoS.
The post was a pretty good explanation about a new distro ISPs can use to help with fair queuing, but this statement is laughably naive.
A distro existing is only a baby first step to an ISP adopting this. They need to train on how to monitor these, scale them, take them out for maintenance, and operate them in a highly available fashion.
It's a huge opex barrier and capex is not why ISPs didn’t bother to solve it in the first place.
We have seen small ISPs get LibreQos running in under an hour, which includes installing ubuntu. Configuring it right and getting it fully integrated with the customer management system takes longer.
We're pretty sure most of those ISPs see reduced opex from support calls.
Capex until the appearance of fq_codel (Preseem, Bequant) or cake (LibreQos, Paraqum) middleboxes was essentially infinite. Now it's pennies per subscriber and many just a get a suitable box off of ebay.
I agree btw, that how to monitor and scale is a learned thing. For example many naive operators look at "drops" as reported by CAKE as a bad thing, when it is actually needed for good congestion control.
How does OpenWRT fair on these metrics? Does it count as a "debloted router" is the sense used in TFA? Or is additional software above and beyond the core OpenWRT system needed to handle congestion properly?
OpenWRT has SQM but you have to enable it. https://openwrt.org/docs/guide-user/network/traffic-shaping/...
OpenWrt depreciated pfifo_fast in favor of fq_codel in 2012, and have not looked back. It (and BQL) is ever present on all their Ethernet hardware and most of their wifi, no configuration required. It's just there.
That said many OpenWrt chips have offloads that bypass that, and while speedier and low power, tend to be over buffered.
Even IT professionals can't tell the difference between latency and bandwidth, or capacity and speed.
A simple rule of thumb is: If a single user experiences poor performance with your otherwise idle cluster of servers, then adding more servers will not help.
You can't imagine how often I have to have this conversation with developers, devops people, architects, business owners, etc...
"Let's just double the cores and see what happens."
"Let's not, it'll just double your costs and do nothing to improve things."
Also another recent conversation:
"Your network security middlebox doesn't use a proper network stack and is adding 5 milliseconds of latency to all datacentre communications."
"We can scale out by adding more instances if capacity is a concern!"
"That's... not what I said."
I share your pain. I really really really share your pain.
It’s astounding how many people that work in infrastructure should understand things like this but don’t, particularly network bottlenecks or bottlenecks in general. I’ve seen situations where someone wants to increase the number of replicas for some service because the frontend is 504’ing, but the real reason is because the database has become saturated by the calls from the service. It is possible (a little unlikely, but possible, and the rule with infra at scale is “unlikely” always becomes “certain”) to actually make the problem worse by scaling up there. The number of blank stares I get when explaining things like this is demoralizing sometimes, especially in consulting situations where you have some pig headed engineer manager that thinks he knows everything about everything.
Being one of the authors of fq_codel, cake and to a small extent, LibreQos, I remain boggled after 15 years of trying, to get these simple points across..
Have some laughs: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-...
As they say, if you’re getting impatient for your baby to arrive, just get more pregnant ladies together! The cluster of pregnant women make the process move along quicker!
/s
And the "they" in question is Warren Buffet, https://nymag.com/intelligencer/2009/06/you_cant_make_a_baby...
I thought this was first attributed to Fred Brooks in the 70s.
> Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, "nine women can't make a baby in one month".
MacOS has now a built in dedicated tool called networkQuality that tries to capture these variables https://netbeez.net/blog/measure-network-quality-on-macos/
Also take a look at Measurement Swiss Army-Knife (MSAK) https://netbeez.net/blog/msak/
big fan of flent.org and this tool, written in rust - is coming along smartly.
There are three parameters of concern with your ISP. Bandwidth, Latency (and Jitter), and Data Caps.
Bandwidth is less of a concern for most people as data rates are over 500mb+ . That's enough to comfortably stream 5 concurrent 4k streams (at 20mbps).
Latency and jitter will have a bigger impact on real time applications, particularly video conferencing , VOIP, gaming , and to a lesser extent video streaming when you are scrubbing the feed. You can test yours at https://speed.cloudflare.com/ . If your video is jittery or laggy, and you are having trouble with natural conversation, latency / jitter are likely the issue.
Data Caps are a real concern for most people. At 1gbps, most people are hitting their 1-1.5tb data cap within an hour or so.
Assuming you are around 500mbps or more, latency & data caps are a bigger concern.
> At 1gbps, most people are hitting their 1-1.5tb data cap within an hour or so.
Assuming you're talking about consumers: How? All that data needs to go somewhere!
Even multiple 4K streams only take a fraction of one gigabit/s, and while downloads can often saturate a connection, the total transmitted amount of data is capped by storage capacities.
That's not to say that data caps are a good thing, but conversely it also doesn't mean that gigabit connections with terabit-sized data caps are useless.
> gaming
Gamers tend to have an intuitive understanding of latency, they just use the words "lag" and "ping" instead.
ACM, come on, stop spreading disinformation. You know well and good nothing travels at the speed of light down the wire or fiber. We have converters on the end, and in fact in glass it is the speed of light divided by the refractive index of the glass. Even in the best of times, not c. I just hate that, when a customer is yelling at me telling me that the latency should be absolute 0, they start pointing at articles like this, "see, even the mighty ACM says it should be c".
Ugh.
And that's before you consider the actual cable length vs the straight line distance.
I don’t think you’re going to have an issue with Cherenkov radiation in the fiber and that fiber is not going to be a straight line over a non trivial distance so the approximation is close enough.
It's a reasonable approximation for most calculations. It seems unfair to call that "disinformation".
Serialization delay, queuing delay etc. often dominate, but these have little to do with the actual propagation delay, which also can't be neglected.
> when a customer is yelling at me telling me that the latency should be absolute 0
The speed of light isn't infinity, is it?
Speed of light also explained why these emails were limited to a little over 500 miles: http://www.ibiblio.org/harris/500milemail.html
So it can be relevant, even as an approximation.