points by est 6 years ago

Some outcomes of this morning:

1. Alibaba's teamwork app, Dingtalk, crashed around 9AM due to too many concurrent video conference saturate the server and bandwitdh

2. Tencent's for enterprise messaging app, Wechat for Business, crashed. Connection is extremely unstable

3. Baidu's office VPN was busy and employees are asked to stay disconnected to leave bandwith for sysadmins

4. Huawei's WeLink was unavailable for a while

5. Bytedance (company behind TikTok)'s Lark, an online office suite like GApps was the biggest winner, only had some minor issues.

6. Zoom offered a free version to mainland users and it's extremely popular. But it lacks non-video-conf features. e.g. simple daily poll to see if your colleagues were healthy or not.

tmp5774889 6 years ago

I've created a temp account for this to be on the safe side :)

I work for a big (300k+ ) company with some tens of thousand office workers in CN We are preparing since about 2 weeks to upgrade our remote access infra in China with partial success as something which would be a soft upgrade everywhere else needs to go through various levels of local subcontractors and partners of our provider

On top of that because of the Internet situation in CN it is not practically easy to use remote access via another location with higher capacity - the performance gets degraded very quickly - or use resources directly over the internet eg. RTC which is not hosted locally

I doubt that local authorities will change something in the future because of this example but one can only hope

  • est 6 years ago

    are you the same guy on r/sysadmin?

    Try use MPLS. It's expensive but the latency and bandwidth is guaranteed.

    • tmp5774889 6 years ago

      nope, not the same guy MPLS is used for the office network anyhow, talking about remote access here

threatripper 6 years ago

This "experiment" will set a new baseline on which management can act with the goal to satisfy such use cases and grant the necessary resources to implement them.

  • koheripbal 6 years ago

    Well the clock is ticking for non-Chinese companies. When SARS struck in 2003, it took about four months for the total global number of cases to crest. Assuming the same sort of pattern here with the same WFH response, we will see peak VPN/remote usage in April/May.

    ....so get those requisitions in boys.

    • diroussel 6 years ago

      A paper published in the Lancet on Friday said that the peak infection is persecuted to be in April. But if more curbs on movement are out in place then the total infections will be lower, but the peak will be later. But this was based on a current infected population of 75k, much higher than the Chinese official figure of 14k people.

      Source Dr John Campbell. https://youtu.be/z05ZrMfKUDc

      • koheripbal 6 years ago

        I think your underlying point is correct - the duration of this outbreak is unknown.

        Regarding the Chinese figure of 14k - it is only for "laboratory confirmed" cases. Given that Chinese hospitals are refusing any patients that aren't in serious distress, the "real" number of cases is likely already much much more than 14k.

        There are also reports of death certificates being issued for "unknown viral pneumonia", because they don't have the time to test the dead.

        This would explain why there are sooo many cases abroad - despite "only" a few thousand cases in China.

        One virologist tweeted (can't find it now), that he believes 95% of cases are unreported in China.

        We also have to bare in mind that most non-western countries do not have DNA test kits for this virus. When we look at the updated map of infections, there are glaring gaps in countries like Laos and Myanmar with ALL of their neighbors reporting infections. Those countries are, of course, not immune to the virus - they simply don't have the facilities to confirm cases, and have gov'ts that would rather hide the evidence so as not to disrupt travel/create panic.

        I don't think we should OVER-react, but also we shouldn't UNDER react.

        • feanaro 6 years ago

          Also, the 14k is over 17k by now.

        • bangboombang 6 years ago

          Imo the model of those Australian scientists from a few days ago forecasting 75k infections in China by Feb. 4th was pretty spot on.

          Official numbers in China right now are 17k infections plus 21k estimated unreported cases.[1] Let's assume their numbers are a little too optimistic and there you go, 75k by tomorrow.

          [1] https://news.qq.com/zt2020/page/feiyan.htm

ekianjo 6 years ago

Seems like the experiment was a success: systems have very low margins to operate.

  • quickthrower2 6 years ago

    We need peer to peer. These central hubs can’t hack the traffic.

    • philbarr 6 years ago

      WebRTC is peer to peer and is supported in most major browsers.

      • yomly 6 years ago

        From experience WebRTC starts to fall down in rooms of more than 8 people for video chat.

        Meanwhile Zoom and Discord (who use servers) can achieve much greater so YMMV

        That said p2p definitely has its advantages

        • dana321 6 years ago

          Its just a case of p2p with people running servers (like bitcoint does) and differentiating between the two based on a bandwidth test.

      • basejumping 6 years ago

        Doesn't WebRTC need a TURN server to proxy traffic when you're behind a NAT? It's not really peer to peer when all the data goes through a central point.

        • johncolanduoni 6 years ago

          TURN only proxies if it’s unable to make a connection via STUN, which requires the server to handle only a few packets when the initial connection is formed and provides direct peer to peer access after. Many home NATs only need the latter.

    • michaelt 6 years ago

      Peer-to-peer has its advantages, but it's no panacea.

      After all, practically everyone is behind NAT these days, and it's so rare for an application to ask you to open a port in your firewall, most users probably don't know how to do it.

      And early videoconferencing was famous for its unreliability - even simple things like people being able to join calls. Particularly as users might be behind restrictive firewalls on corporate networks or cell phone connections. Video calling that only works 95% of the time isn't good enough for things like job interviews :)

      And users don't only expect one-to-one calls - everyone offers multi-party calls. If eight people want to watch Bob while he's talking, you've got to find eight videos' worth of upload bandwidth from somewhere.

      And there are far more people trying to video call from battery-powered devices and metered connections than there were 15 years ago. If your competitors' apps uploads one video stream and yours uploads eight at once, users are going to notice the battery and data consumption.

      And if you only have to test compatibility between your server and every version of your software, that's a much simpler task than testing compatibility of every possible n-way call between different versions of your software. Especially as you'll have to support Windows, OS X, Android and iPhone at the very least.

      • the8472 6 years ago

        You could still do opportunistic P2P to lessen server load. And NAT is an issue, but not insurmountable, perhaps both sides have IPv6 or one side has a router with UPnP, PCP or a NAT implementation that makes hole punching easy.

      • dana321 6 years ago

        Thats a pretty negative picture you paint there.

        To me, those are fun things to solve.

        The answer is to still have clients and servers, just that the servers deal with the grunt work of routing traffic and the clients just stream the data through a server.

        Push vs pull, depending if you can get upnp to open a port to the client.

        Skype was originally a peer to peer application, seemed pretty popular and while it still was p2p? It was written by the people who wrote Kazaa, remember that?

        • syshum 6 years ago

          and one of the primary reasons Skype is no longer p2p is to violate user privacy not because of technical reasons

          • amaranth 6 years ago

            I think Ars Technica's take on this makes the most sense. Basically the new architecture might make violating privacy easier but they could do that with the old one too and there are legitimate improvements from the new architecture that are more than enough explanation for the switch.

            https://arstechnica.com/information-technology/2016/07/skype...

            • syshum 6 years ago

              The main reason was to make is compliant with the desire of US Law Enforcement. but yea continues to believe what you want. I really dont care

              All of the performance improvements could have been done while maintaining the core of P2P, the ONLY reason to centralize it is to make it easy to tap, and harvest data from

        • michaelt 6 years ago

          *> Skype was originally a peer to peer application&

          Sure, but Skype adopted a peer-to-peer design back in 2005 or so when expectations were a lot lower - no smartphones, no group calls, no 1080p, no cross-platform, and if it's unreliable or needs some fiddling around to get it to work properly, such is life - because the competition was paying $$$ for an international phone call that wouldn't have any video.

          These days if you want to compete with Skype and Google Hangouts and Discord, I expect my regular four-way video call between people using Windows, OS X, Ubuntu and ChromeOS to establish first-time with clear audio and video to all users.

          • mehhh 6 years ago

            If anything, P2P high bandwidth connections are often more reliable as your traffic likely stays on your ISPs network rather than needlessly hairpinning through a central server outside their network.

            Hence why VoIP like WebRTC encourages direct media rather than wasting a trip to a server, the server's peering can often be much worse than a path between two users on residential ISPs.

            • frandroid 6 years ago

              For 2 users, fine. But the parent was mentioning 8 users. You can't have live conversation delays/asynchronicity that p2p'ing through multiple users would bring.

      • slovenlyrobot 6 years ago

        > it's so rare for an application to ask you to open a port in your firewall, most users probably don't know how to do it.

        in home settings this step has been automated for something like a decade or more. One of UPnP or NAT-PMP are available on basically every router

        • heinrichhartman 6 years ago

          But are they enabled and working in the 99.5% house-holds, who do not have a professional Sys-Admin at home?

        • pixl97 6 years ago

          UPnP is pretty much a security disease. Countless security holes have come from that protocol. Worse is tons of users are behind double NATs and carrier NAT.

          Really we just need IPv6 where one app binds to one IP.

      • wbl 6 years ago

        IPv6 is ancient at this point and we still don't have it deployed.

        • xvilka 6 years ago

          It's widely deployed in China - in households, Web and mobile services and applications.

          • kccqzy 6 years ago

            I find that hard to believe. The last few times I've been to China (in 2019), none of the hotels I stayed in (all of them in Shanghai) had ipv6 for their guest WiFi.

            • xvilka 6 years ago

              I think it's a problem with the legacy infrastructure of WiFi hotspots in the hotels instead. And the final steps of the deployment were in October 2019, I think. Most mobile applications, China Mobile and China Telecom, both landline and mobile connections - everything works with IPv6 now. Even big corporate networks started to do the switch.

        • sliken 6 years ago

          Over 1/3rd of Google's US traffic is IPv6, so it's not exactly rare.

        • satanspastaroll 6 years ago

          IIRC IPv6 became a ratified internet standard in 2017

      • satanspastaroll 6 years ago

        Here's a wish for rapid adoption of IPv6, where no one will be fored to use nat ever again

        >If eight people want to watch Bob while he's talking, you've got to find eight videos' worth of upload bandwidth from somewhere. With p2p you may stream parts of the video from multiple clients

    • michaelcampbell 6 years ago

      But then your data can't be taken and tracked and sold. Never happen.

    • MockObject 6 years ago

      If only we hadn't killed multicast.

tener 6 years ago

It would be interesting to know where are the bottlenecks and how quickly they can be addressed.

  • onetimemanytime 6 years ago

    I suspect that they might be addressed in time for the next epidemic

wyxuan 6 years ago

That's good to hear. Especially that last point: zoom has been repeatedly blocked by China. It would be interesting to see if they could keep those users

  • est 6 years ago

    IIRC Zoom released a tailored "China" version.