I hate big power. When I was an IBM engineer, power into the computer room was a big deal. Big diesel generators feeding into the main switchboard along with redundant mains connections.
One incident I remember is when an electrician was working on one of the power feeds while it was live. Foolish, but the kind of thing people did when getting approval to shut off power was a bureaucratic nightmare.
The sparky was tightening down a bolt on one of the phases with a big crescent spanner when he inadvertantly made contact with another phase.
There was a flash that lit up the room and a loud bang. His crescent spanner literally melted in the middle. Incredibly he was uninjured, although he did take the whole room down, which was an extremely serious matter.
Modern computers that plug into the wall socket are much more pleasant to work on.
Nice, my favorite big power story was a technician who was bringing up the next set of "rows" in a very large web scale data center.
One component of that task was running thousands of feet of CAT-6e cable. Of course you want to be sure your spool of CAT-6e cable has no wire breaks in it before you start using it[1] to wire up racks so there was a step that involved putting a connector on the stub of cable that was inside the reel, and a connector on the end of the cable that came off the reel, and then hooking up an instrument between them to measure the resistance/propagation/etc.
It is convenient to work at waist height for this, so our hapless technician had the cable spool put on a convenient big square brick like metal structure outside the building. He added the connectors, hooked up the tester, and the tester exploded. (well not completely but it did experience some rapid unplanned disassembly due to a large number of capacitors suddenly not existing.
The "brick like structure" that the technician was using as a table top happened to be one of the transformers that was converting high voltage down to 440V 3-phase for the data center. When he effectively shorted the cable loop with his tester he air coupled into the field around the transformer and put > 400V into the tester. Whoops!
[1] tracking down a bad network cable is not a lot of fun.
This is why (at least in my country) there are very clear codes about delineating power and data equipment. When we run data, it’s always at least a meter from any power cables or equipment.
That is an excellent policy, do you have any companies in your country where the company believes itself to be really really smart and also believes rules like this are just bureaucracies trying protect their own jobs?
It had the required shielding and warnings about opening the box. That said, the spool of cable was likely 2500' of CAT-6e cable. In the root cause analysis it was estimated the voltage potential on the cable was between 210 and 280V and the current levels could have reached 500mA (a bit more than 100W of 'leakage'). That was well outside the what the test gear was "expecting" from a cable that it was testing.
Electricians work on live stuff all the time. They just rarely do so in residential settings which is why most people don't realize that. He was just sloppy. There procedures you're supposed to be following (like using insulated tools) to prevent stuff like that.
Probably the idea was to keep the current from crossing the heart/chest area, the right side being further away (to some extent in most people).
There's a similar rule for HV work (and when working with vacuum tube-based equipment) - "one hand in pocket"; again, the idea being that if you're working on a live system, you aren't touching things with both hands leading to a shock across the heart area.
Another rule for live systems (this applies both for higher voltages and for low voltage systems - it's mainly to protect the circuit and components):
If you have to test/measure a live system, it's best to hook up the test equipment in the "dead" state, then power up the system, then shut down before removal or changing the position of the test leads, etc.
But if this isn't possible, the next best thing to do is to hook one lead (if say using a multimeter) to a common reference point (that depends on the circuit and what you're measuring and where - but usually it's a shared ground point), and probe with the other using only one hand. Try to never probe with both hands/probes - because it can lead to inadvertent shorts and other problems. As always, there are exceptions to this rule, but they should only be considered after very careful thought and planning.
I'm generally in awe of long-lived procedures intended for mass distribution.
There's a beauty of succinct purpose.
A 14 year old with no scientific background can understand "one hand in pocket or you're dead." He or she would far less reliably understand "This is how voltage works. This is how it reacts with your body. This is how..."
Good thing tools are made of such resistive metals. I can only imagine how that would have gone if the metal with the best tensile strength for a wrench was copper.
Some wrenches are in fact made of copper with about 1% beryllium; it's used because it does not create sparks when hit, useful in places where sparks are dangerous like a natural gas plant for example. More info: https://en.wikipedia.org/wiki/Beryllium_copper
My senior year physics teacher in high school was a PhD student at Princeton who was doing teaching waiting to get back to his doctoral thesis research.
Apparently, his work was with the Plasma Lab, primarily the Tokamak they had. Someone at some point had dropped a wrench or something on the bus bars of the capacitor bank used to power it - and, well, that was that. So he couldn't continue his studies until it was repaired.
In the meantime, he taught us.
Strange guy, but insanely intelligent, and fun to talk with. He always stressed the importance of "doing the experiment" - so we were always doing a practical experiments as a group to show how what we were shown and learned in class actually worked (within margins of error of course). I learned a ton in that class.
After I graduated, I think he taught there one more year, then a few years later I looked him up, and he was back at Princeton, but was now working on something having to do with Materials science; I'm not sure what he's up to today, though - that was a couple decades ago. He wasn't much older than us at the time; late 20s or so.
If anyone is in Victoria, Australia and wants to play with an IBM z9 mainframe, i picked one up for about $500 and although I've managed to power and boot it, i haven't managed to load zOS on it properly. Got some time to spend on it and would love it if anyone else is interested in experimenting/getting it up and running.
I also bought the relevant storage array (but no disks). Unfortunately that takes up a heap of space and I'm looking to offload it, it's free if anyone wants it, otherwise ill probably have to scrap it :(
I usually lurk here but feel free to reach out via twitter (same username)
I’ve seen those “support element” laptops hanging off of racks before, but I always assumed that in this case (IBM mainframe, IBM support element, vertical integration) they were not laptops as such, but rather thin clients in the most literal sense: just IO interfaces with a ribbon cable feeding back to the mainframe, where the framebuffer of the mainframe’s dom0/maintenance VM was pushed over VGA and keyboard+mouse+peripherals were returned over USB. Essentially as if the support element were a Thunderbolt Display with USB HID devices plugged into it. Like a modern VT100!
I’m honestly surprised to learn that I’m wrong, as having the laptop be a real laptop would introduce all sorts of headaches compared to having the laptop be a dumb terminal—it’d have active components like disks and fans that could to burn out (and not be hot-swappable like the server’s own components are); and more importantly, it’d have to talk to the server using some form of network interface (unless it’s just a glorified glass terminal for a USB-serial adapter.) In which case, bad behaviour on the server—exactly the case where you need a local admin console—could block or steal resources from your connection attempt. You may as well just be SSHing into the dom0 from across the room over the SBC VLAN at that point.
The support laptop on an IBM mainframe is analogous to the setup on a Cray machine. The System Management Workstation or 'SMW' (previously a Supermicro tower workstation with a bunch of ethernet), now a Dell Poweredge rackmount is its own system which controls the rest of the machine--booting, shutdown, and all of the other myriad maintenance tasks on a cluster like that. It's also responsible for building boot images and interrogating the low level controllers on each rack and on system blades themselves.
During bringup of a machine like that, the SMW host is installed first from scratch with a Linux distribution, hardware inventory is enumerated, boot images are built for individual node roles in the system and then the system performs its boot procedure.
The new iSeries we bought (Power S914) uses a Windows computer as its terminal. I would prefer a dedicated terminal with all the keys, but that how it goes now. One thing about it, the tech that is doing the installation is remoted into that PC doing the steps to copy our old machine to the new one.
On a side note, the new machine has LTO-7 and our old machine (9406-800) predates LTO, so it makes the upgrade a bit longer.
Support element has to be more or less full-blown computer as the mainframe itself does not have any real firmware to boot^H^H^H^HIPL itself. Before support elements the IPL process involved initalizing various things by toggling switches on the console.
Also why the mainframe would even have a framebuffer?
Providing a console interface (ie. 3270 terminal emulator) to whatever OS runs on the mainframe is only small part of what SE does.
> Also why the mainframe would even have a framebuffer?
The mainframe itself wouldn't, but it might be running a VM that has one (and also has reserved capacity on the system, preventing other tasks from fighting it for priority.) I was picturing the support element as simply a physical extension of that VM, sort of like x86 VMs can use IO-SRV to lay claim to graphics cards and such. It'd be like the BMC on an x86 server motherboards, but virtual.
What you say about the support element also being used to IPL the system makes sense, but I'm surprised that responsibility is pushed into such a fragile component as a laptop. Why not just make the "IPL booter" a single-responsibility SoC card-edge daughterboard (I'm picturing a game-cartridge-looking thing), and leave the support element for, well, support?
Just a word of caution for anyone reading this. I highly discourage anyone from actually learning ^w. I learned it, then had to forcefully un-learn it when I kept accidentally closing tabs on my browser, other people's browsers, my file manager, etc. etc. ad nauseam. Don't make the same mistake I did, just press backspace a lot. I know life is short but the sudden rage of having unexpectedly closed a tab is just not worth the added efficiency. I am passionate about this subject; however, I mean no disrespect for those who have (somehow) managed to use ^w productively.
I like seeing posts like this showing that there is still interest in operating with mainframes and bare-metal servers these days.
My biggest fear is that you are in the minority of developers that can single-handedly operate one in the age of buzzword-ridden lingo such as AWS, GCP, Heroku, etc which forces a dogma to run rampant in our industry to host all of your startup/company only on other peoples's servers rather than to setup up your own in-house bare-metal servers instead.
So I am very impressed to see this, as an added bonus it is a mainframe. Please post more of this.
It's not about clouds. The problem is with mainframe manufacturers. Their prices are ridiculous, so no sane man with limited budgets would buy it. I can build incredible powerful server from a desktop or workstation components for a few thousand dollars. Probably can reduce it to hundreds with using old parts. I can increase price to x2 and buy some HP or Dell blade and that would be real server hardware. But nobody is going to increase price to x100 and buy mainframe. That's just too much.
Keep in mind that a lot of the people that are running mainframes now have been running them for many years (think 1980s), and there was no alternative that could run at the scale / speed / throughput that the mainframe can. So much of what we do today in the cloud can be attributed to the trails blazed by the mainframers. Fascinating technology.
> My biggest fear is that you are in the minority of developers that can single-handedly operate one
You're right. About 6 months ago the University of Canberra, Australia started it's 'Bachelor of Information Technology in Mainframe Computing' degree program because of the shortage.
This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really high-capacity x86 machine? I know that one difference is that you can hot-swap pretty much any component on a mainframe, but would be very curious to learn more about these beasts.
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus.
Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
The other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc).
An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.
"Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.
I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things).
While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one.
What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way.
They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption.
So basically what I got from that is that the benefits are:
There's literally no such thing as 100% uptime. You can add more "nines", meaning that disruptions become exponentially less likely, but that's still very different from "100%".
I think '100% uptime' is pretty much shorthand for "absolutely no reason to turn the machine off or stop services on them running, ever".
As the above link mentions, everything is redundant. CPUs running in (near) lockstep - voting bad cpus out, redundant storage, power etc etc etc.
I've not seen a better write up on IBM's equivalent systems - but I've not had a reason to play with such big iron. NSKs however were much more affordable, and were used in the late 90s/early 2000s in various telcos etc.
I mean, a meteor could hit the DC. But I guess that would count as a decommissioning event, wouldn’t it? So you’re still right, with a proviso—there’s never any reason to turn the machine off with the expectation of ever turning it back on.
A company I worked for (in São Paulo) had a couple failover setups for its Unisys A-series machine, one with a bank across the street and another with a near twin machine running in its factory in Manaus.
If, for some reason, an event decommissioned all three machines, having our computers back online would be a lesser problem - we would probably be better off learning to hunt and make fire.
But why ? Other than inertia and legacy reasons that is. Is there some non partisan resource on this ? A paper or a neutral blog ? Whenever you try to research something like this, you end up on either IBM pages or pages by mainframe enthusiasts. I want to know exactly what you are buying compared to a regular high availability database like setup in the cloud.
I work fairly regularly with our mainframe team, and the general reasoning boils down to massive parallelism and consistency when processing critical data. The number of rows our mainframe can process and write with perfect accuracy is leagues ahead of what similarly priced commodity hardware could achieve. There’s a reason the world still pretty much runs on COBOL/Natural and a hardware design from the 70s, it worked well then and has worked well ever since. Slightly related but I’d describe most of my role being getting data from our micro-services down into the mainframe in a format it understands so we can do ‘something’ with it.
Interesting. So something like google bigquery (or any of the various saas offerings by others) would not be a good match for this problem space ? For massive parallelism, we have GPUs, accelerators, and database products built on these things (admittedly, still somewhat new). By consistency, do you mean tolerance to transient errors and the like ? i.e. Beyond whatever you get with ECC ? Or are you talking about memory consistency models ?
I am just constantly surprised that on the one hand FAANG+ which run so much of the internet economy do not use mainframes, and yet banks do.
The promise of the cloud is that it provides you with "trivially", "infinitely", "cost-efficiently" scalable infrastructure. Banks don't need any of that. They can buy a $100k (?) mainframe, a hundred or so devs, and leave that part of the business on autopilot for 20+ years. Why even bother futzing around in the cloud?
Getting distributed systems correct is very hard. Mainframe architecture allows one to program and deploy as if you had a giant computer with consistent transactions for the most part. Since banks were the early adopters of computation in large scale it’s clear why most still run on mainframe - the transition to cloud architecture isn’t a simple translation, you need to glue a lot of things together to the same levels of parallelism, and probably sacrifice some consistency that you get “for free” on mainframe architecture.
Cloud is cheaper ($ per cycle) at the expense of human brain power to glue it all together.
In addition to what bendbro suggests, which is actually quite close to reality if oversimplified, banks also have sovereignty concerns with data that may have additional classification and regulatory strings attached. In Australia where I’m based, ‘material’ data such as financial and audit information must be kept in country and its veracity attested to, which is why so many large banks here at least still have big iron in managed data centres for most of their core transactional systems.
Edited to specify which sibling I was referring to.
Outsourcing of financial services processing has been a big business for decades. However it is usually provided to small/medium banks or credit unions by specialized firms or by big banks.
A big bank will usually do its own ledger processing. A good configuration is to process in two pairs of data centers more than 200 miles apart in different electric utility interconnection regions, e.g. Arlington/Fort Worth and Dayton/Springfield. Archival storage of transactions may involve additional centers.
https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex#Geographi...
Give me a message queue that guarantees exactly-once-delivery to coordinate a cluster of transaction processing mashines and we can talk about banks using commodity hardware.
Someone posted in a related thread I was nerding out on tonight that mainframes are excellent when you need to do an enormous amount of I/O with 100% uptime and "exactly once" consistency guarantees. According to that poster, you also don't need to build distributed logic or certain kinds of error handling into your software.
My high level view is that with a mainframe you get a lot of the advantages of the cloud without the hassle of implementing a distributed system / data integrity and consistency over the network (ie something like having the AWS console where you can instantiate services but everything is local. Upgrading the account is physically putting more/swapping resources in the Mainframe). Also there is the cost of migrating, these systems work perfectly reliably since more than 30 years.
It goes a bit deeper than that. I work on POWER which is a sibling to the z platform. They share some common architecture and microarchitectural elements.
To get a sense, have a look at the E980’s Reliability, Availability, and Serviceability Manual. [1] that systems NOT a mainframe - but it borrows ideas from it. Even in that system all levels of it are designed for failure, from Customer replaceable units like fans or power supplies to certain busses have extra lanes or wires for parity or spares depending on the design. Internal to the processor logic there’s a large number of internal checkers that watch the architected state of the machine and notice when something abnormal happens. For several things the processor core will unwind its state and retry an op, and if it proceeds, flag itself as having had a “recoverable” error, which usually triggers a service action. If the system can’t recover it halts either the core or the entire platform, and a dump happens from the service processor which then offloads it to the management console for diagnosis.
Z boxes do all of the above - and more. There’s pictures floating around the office of a system that went through and earthquake, fell over (someone didn’t buy the optional earthquake bracing for the rack) and as it fell, it domino’d over along with several other racks. It still running when the techs went to inspect the data center, though it did turn on its “service me” attention light. ;)
That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.
I looked at the author's previous post where he details the process of buying the thing. A fair amount of it goes over my head, I must admit. But that final price tag.... You've gotta really love mainframes. I can't imagine the utility bills for it. The power usage was reduced to 1.7kW. I guess some people buy a sports car and some buy a mainframe. Personally, I'd never be able to drop that kind of money for a hobby. I think emulating a mainframe on a fancy PC would scratch my itch. But to each their own.
If you asked me how much I thought 700kg of computer should cost, I probably would have guessed more than $12k. Is this thing modern? Specs don't seem too impressive but I don't know anything about these things.
I was about to think that was old, but then I remembered that mainframes, as I understand it, can aim for decade+ uptime, so that might actually be reasonably fresh?
Interesting it runs straight off normal 3phase mains - the only mainframe I worked (a bit ) on was a small Unisys one and it had its own 400v supply as part of the set up.
The Bulk Power Regulator mentioned in the blog post is essentially an 400V DC supply that powers everything else in the rack (including the blowers with threephase motors and their own DC/AC invertor).
So it is similar except the power supply is built on modern power electronics and is quite small and internal in the rack.
>The Bulk Power Regulator mentioned in the blog post is essentially an 400V DC supply that powers everything else in the rack (including the blowers with threephase motors and their own DC/AC invertor).
Any decent motor these days is a brushless DC motor, so it runs on DC voltage, except that it really doesn't. It gets DC voltage from a power supply (or it rectifies AC voltage to get DC, just like your computer's power supply), and then uses that to drive a motor drive (controller) (a box of electronics that runs the motor). The motor drive generates 3-phase AC power which is actually used to power the windings of the physical motor. The motor includes feedback devices which tell the controller the motor shaft's position so it knows where in the AC cycle the output waveform should be.
There's always a lot of confusion about these motors because of this: they're called "brushless DC" (BLDC), but also 3-phase AC, and both are correct. The key is that the controller is really an integral part of the motor. It's similar to how your CPU probably runs on 1.8VDC, but you don't actually supply it that from your computer's power supply, the motherboard has its own voltage regulator to create that. You would never actually make a separate 1.8V power supply for a CPU, because it wouldn't work very well (high losses over distance); it's much more efficient to have the regulator closely coupled to the CPU.
> Any decent motor these days is a brushless DC motor
Most applications are perfectly fine with asynchronous motors which don't require extra power electronics. The increase in efficiency from IE3 asynchronouos to IE4 synchronous/SRM/hybrid is pretty small and costs a bunch.
Depends on what it is. Even the little fans in desktop computers now are BLDC now, with the controller fitting on a tiny PCB inside the hub of the fan. They're simpler versions, though, with hall-effect feedback rather than an encoder. The power electronics these days are dirt cheap.
> Even the little fans in desktop computers now are BLDC now
Those are at the lower end of the power spectrum of electric motors (which spans approximately 10 orders of magnitude from a few mW to hundreds of MW) and have been BLDCs for a very long time. To see something else in a PC you probably have to go all the way back to the IBM PCs.
> The power electronics these days are dirt cheap.
For a <1 W fan, yes, but when you want to run a 5.5 kW saw it still costs real money and the motor is more expensive.
The reason why I specifically mentioned the blower motors is because the blowers in most mainframes I have seen have use small(-ish) industrial 3phase synchronous motors, not purpose-designed BLDC motors.
I was a former mechanical maintenance tech for an auto manufacturer, so this story feels familiar to connecting a CNC lathe, or a multi-axis mill, but reading this guys ham-fisted accounts made me pretty livid..
>The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.
Since this thing can approach the cost of a rolls-royce,and weighs half as much as a prius, movers and millwrights should have been called in. watching two chubby guys in tennis shoes and no gloves move a 700 kilo server in reverse would have given me a heart attack.
> The various guides did hint towards how the phases are connected to various Bulk Power Regulators (BPR) but nothing was very definite.
that should have been your clue to hire an electrician. there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR.
>I tried to do my best to construct a power cable
Make sure you tell your building supervisor and the fire brigade you "tried your best." there is no "tried your best" section in the electrical code. You either construct a cable to spec, or not.
>Now, I assumed that given there is no neutral it has to be a delta system
Since youre handling voltages that can kill you, perhaps a meter?
>The uncertainty was which phase-pair powered the lowest BPR? I guessed A-B
OR your BPR explodes, your HV breaker trips, and you wipe out power for half the building. Christ on his throne.
> there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR
No, in IBM mainframes, the rack component that takes the mains AC and starts doing something with it is, indeed, called a "bulk power regulator". For example, IBM part number 6186-7040, "Bulk Power Regulator".
> that should have been your clue to hire an electrician
Note in the article he did say "I confirmed the wiring scheme with some friends I have that do the power installations at Dreamhack, and even had them do the wiring as they have all the qualifications needed to do so safely and legally."
Why? It shouldn't be. They use 3-phase in Europe just like in the US, for industrial locations. The differences in residential or lower-voltage standards really have nothing to do with industrial power availability.
The author described some shenanigans he had to go through when wiring the power cord to get the power supplies (which expected voltage that is typical in the US) to run right. In the US it would have been a simple matter of using a power cord in the standard configuration and plugging it into the wall.
Looking closely at the photo of the power supply in the article, it looks like it's quite flexible and can handle different voltages, as long as they're 3-phase. It even lists different amp draws for the different voltage levels. In addition, it looks like it can handle 380-570V DC. With today's switching power supply technology, it's not hard to make them able to accept widely differing input voltages, so this is commonly done so you only need one part to cover the entire worldwide market. It looks to me like you can just plug this thing into any 3-phase power you want really.
As for a "standard configuration" in the US, there really isn't such a thing for 3-phase power. There are a few NEMA standards used in industrial applications, but a lot of times, things like that are hard-wired. This isn't a home appliance.
I hate big power. When I was an IBM engineer, power into the computer room was a big deal. Big diesel generators feeding into the main switchboard along with redundant mains connections.
One incident I remember is when an electrician was working on one of the power feeds while it was live. Foolish, but the kind of thing people did when getting approval to shut off power was a bureaucratic nightmare.
The sparky was tightening down a bolt on one of the phases with a big crescent spanner when he inadvertantly made contact with another phase.
There was a flash that lit up the room and a loud bang. His crescent spanner literally melted in the middle. Incredibly he was uninjured, although he did take the whole room down, which was an extremely serious matter.
Modern computers that plug into the wall socket are much more pleasant to work on.
Nice, my favorite big power story was a technician who was bringing up the next set of "rows" in a very large web scale data center.
One component of that task was running thousands of feet of CAT-6e cable. Of course you want to be sure your spool of CAT-6e cable has no wire breaks in it before you start using it[1] to wire up racks so there was a step that involved putting a connector on the stub of cable that was inside the reel, and a connector on the end of the cable that came off the reel, and then hooking up an instrument between them to measure the resistance/propagation/etc.
It is convenient to work at waist height for this, so our hapless technician had the cable spool put on a convenient big square brick like metal structure outside the building. He added the connectors, hooked up the tester, and the tester exploded. (well not completely but it did experience some rapid unplanned disassembly due to a large number of capacitors suddenly not existing.
The "brick like structure" that the technician was using as a table top happened to be one of the transformers that was converting high voltage down to 440V 3-phase for the data center. When he effectively shorted the cable loop with his tester he air coupled into the field around the transformer and put > 400V into the tester. Whoops!
[1] tracking down a bad network cable is not a lot of fun.
This is why (at least in my country) there are very clear codes about delineating power and data equipment. When we run data, it’s always at least a meter from any power cables or equipment.
That is an excellent policy, do you have any companies in your country where the company believes itself to be really really smart and also believes rules like this are just bureaucracies trying protect their own jobs?
With that much power floating around, I would have expected the transformer to have better shielding/warnings. Or is that not practical?
It had the required shielding and warnings about opening the box. That said, the spool of cable was likely 2500' of CAT-6e cable. In the root cause analysis it was estimated the voltage potential on the cable was between 210 and 280V and the current levels could have reached 500mA (a bit more than 100W of 'leakage'). That was well outside the what the test gear was "expecting" from a cable that it was testing.
I'd guess the tech was probably already in a power shed / access limited area.
And given the distance scaling, it wouldn't have been a problem with a few more inches. He essentially lucked into the worst possible setup.
Electricians work on live stuff all the time. They just rarely do so in residential settings which is why most people don't realize that. He was just sloppy. There procedures you're supposed to be following (like using insulated tools) to prevent stuff like that.
Also relevant: https://i.imgur.com/HjqfZZh.jpg
Plus, some have to work in really unusual and dangerous situations http://www.graybaresp.com/helicopter-lineman/
My ex-Navy middle school science teacher apparently used to work on submarine DC systems.
Per him, you worked with your right hand only and your right foot bare. Better to crisp your right side than send current through your left.
Probably the idea was to keep the current from crossing the heart/chest area, the right side being further away (to some extent in most people).
There's a similar rule for HV work (and when working with vacuum tube-based equipment) - "one hand in pocket"; again, the idea being that if you're working on a live system, you aren't touching things with both hands leading to a shock across the heart area.
Another rule for live systems (this applies both for higher voltages and for low voltage systems - it's mainly to protect the circuit and components):
If you have to test/measure a live system, it's best to hook up the test equipment in the "dead" state, then power up the system, then shut down before removal or changing the position of the test leads, etc.
But if this isn't possible, the next best thing to do is to hook one lead (if say using a multimeter) to a common reference point (that depends on the circuit and what you're measuring and where - but usually it's a shared ground point), and probe with the other using only one hand. Try to never probe with both hands/probes - because it can lead to inadvertent shorts and other problems. As always, there are exceptions to this rule, but they should only be considered after very careful thought and planning.
I'm generally in awe of long-lived procedures intended for mass distribution.
There's a beauty of succinct purpose.
A 14 year old with no scientific background can understand "one hand in pocket or you're dead." He or she would far less reliably understand "This is how voltage works. This is how it reacts with your body. This is how..."
Not wanting to do paperwork generally isn't a good enough reason to do live work, though.
Good thing tools are made of such resistive metals. I can only imagine how that would have gone if the metal with the best tensile strength for a wrench was copper.
Some wrenches are in fact made of copper with about 1% beryllium; it's used because it does not create sparks when hit, useful in places where sparks are dangerous like a natural gas plant for example. More info: https://en.wikipedia.org/wiki/Beryllium_copper
Proper computing , I have heard similar war stories about telecoms accidents with large dc power systems and shorting the busbars.
Real Datacentres have diverse routing for every thing
My senior year physics teacher in high school was a PhD student at Princeton who was doing teaching waiting to get back to his doctoral thesis research.
Apparently, his work was with the Plasma Lab, primarily the Tokamak they had. Someone at some point had dropped a wrench or something on the bus bars of the capacitor bank used to power it - and, well, that was that. So he couldn't continue his studies until it was repaired.
In the meantime, he taught us.
Strange guy, but insanely intelligent, and fun to talk with. He always stressed the importance of "doing the experiment" - so we were always doing a practical experiments as a group to show how what we were shown and learned in class actually worked (within margins of error of course). I learned a ton in that class.
After I graduated, I think he taught there one more year, then a few years later I looked him up, and he was back at Princeton, but was now working on something having to do with Materials science; I'm not sure what he's up to today, though - that was a couple decades ago. He wasn't much older than us at the time; late 20s or so.
> Incredibly he was uninjured
Usually people who make that mistake don't get away any where close to that cleanly. One of the more horrific ways to accidentally die...
If anyone is in Victoria, Australia and wants to play with an IBM z9 mainframe, i picked one up for about $500 and although I've managed to power and boot it, i haven't managed to load zOS on it properly. Got some time to spend on it and would love it if anyone else is interested in experimenting/getting it up and running. I also bought the relevant storage array (but no disks). Unfortunately that takes up a heap of space and I'm looking to offload it, it's free if anyone wants it, otherwise ill probably have to scrap it :(
I usually lurk here but feel free to reach out via twitter (same username)
I’ve seen those “support element” laptops hanging off of racks before, but I always assumed that in this case (IBM mainframe, IBM support element, vertical integration) they were not laptops as such, but rather thin clients in the most literal sense: just IO interfaces with a ribbon cable feeding back to the mainframe, where the framebuffer of the mainframe’s dom0/maintenance VM was pushed over VGA and keyboard+mouse+peripherals were returned over USB. Essentially as if the support element were a Thunderbolt Display with USB HID devices plugged into it. Like a modern VT100!
I’m honestly surprised to learn that I’m wrong, as having the laptop be a real laptop would introduce all sorts of headaches compared to having the laptop be a dumb terminal—it’d have active components like disks and fans that could to burn out (and not be hot-swappable like the server’s own components are); and more importantly, it’d have to talk to the server using some form of network interface (unless it’s just a glorified glass terminal for a USB-serial adapter.) In which case, bad behaviour on the server—exactly the case where you need a local admin console—could block or steal resources from your connection attempt. You may as well just be SSHing into the dom0 from across the room over the SBC VLAN at that point.
The support laptop on an IBM mainframe is analogous to the setup on a Cray machine. The System Management Workstation or 'SMW' (previously a Supermicro tower workstation with a bunch of ethernet), now a Dell Poweredge rackmount is its own system which controls the rest of the machine--booting, shutdown, and all of the other myriad maintenance tasks on a cluster like that. It's also responsible for building boot images and interrogating the low level controllers on each rack and on system blades themselves.
During bringup of a machine like that, the SMW host is installed first from scratch with a Linux distribution, hardware inventory is enumerated, boot images are built for individual node roles in the system and then the system performs its boot procedure.
The new iSeries we bought (Power S914) uses a Windows computer as its terminal. I would prefer a dedicated terminal with all the keys, but that how it goes now. One thing about it, the tech that is doing the installation is remoted into that PC doing the steps to copy our old machine to the new one.
On a side note, the new machine has LTO-7 and our old machine (9406-800) predates LTO, so it makes the upgrade a bit longer.
Support element has to be more or less full-blown computer as the mainframe itself does not have any real firmware to boot^H^H^H^HIPL itself. Before support elements the IPL process involved initalizing various things by toggling switches on the console.
Also why the mainframe would even have a framebuffer?
Providing a console interface (ie. 3270 terminal emulator) to whatever OS runs on the mainframe is only small part of what SE does.
> Also why the mainframe would even have a framebuffer?
The mainframe itself wouldn't, but it might be running a VM that has one (and also has reserved capacity on the system, preventing other tasks from fighting it for priority.) I was picturing the support element as simply a physical extension of that VM, sort of like x86 VMs can use IO-SRV to lay claim to graphics cards and such. It'd be like the BMC on an x86 server motherboards, but virtual.
What you say about the support element also being used to IPL the system makes sense, but I'm surprised that responsibility is pushed into such a fragile component as a laptop. Why not just make the "IPL booter" a single-responsibility SoC card-edge daughterboard (I'm picturing a game-cartridge-looking thing), and leave the support element for, well, support?
> real firmware to boot^H^H^H^HIPL itself.
Have you considered using ^W?
Just a word of caution for anyone reading this. I highly discourage anyone from actually learning ^w. I learned it, then had to forcefully un-learn it when I kept accidentally closing tabs on my browser, other people's browsers, my file manager, etc. etc. ad nauseam. Don't make the same mistake I did, just press backspace a lot. I know life is short but the sudden rage of having unexpectedly closed a tab is just not worth the added efficiency. I am passionate about this subject; however, I mean no disrespect for those who have (somehow) managed to use ^w productively.
No ^W in Emacs nor ISPF ;)
All of which would require the customer to make one of those lovely support calls and create opportunity for upselling!
I like seeing posts like this showing that there is still interest in operating with mainframes and bare-metal servers these days.
My biggest fear is that you are in the minority of developers that can single-handedly operate one in the age of buzzword-ridden lingo such as AWS, GCP, Heroku, etc which forces a dogma to run rampant in our industry to host all of your startup/company only on other peoples's servers rather than to setup up your own in-house bare-metal servers instead.
So I am very impressed to see this, as an added bonus it is a mainframe. Please post more of this.
It's not about clouds. The problem is with mainframe manufacturers. Their prices are ridiculous, so no sane man with limited budgets would buy it. I can build incredible powerful server from a desktop or workstation components for a few thousand dollars. Probably can reduce it to hundreds with using old parts. I can increase price to x2 and buy some HP or Dell blade and that would be real server hardware. But nobody is going to increase price to x100 and buy mainframe. That's just too much.
Keep in mind that a lot of the people that are running mainframes now have been running them for many years (think 1980s), and there was no alternative that could run at the scale / speed / throughput that the mainframe can. So much of what we do today in the cloud can be attributed to the trails blazed by the mainframers. Fascinating technology.
IBM is dead until it gets a mainframe in AWS region data centers. Mainframe is still king of CA in CAP.
Last time I checked IBM was alive and doing well.
They’ve been in a 10+ year turnaround. “Doing well” is debatable.
The way I see it, cloud containers boil down to extending Java JEE like facilities to other stacks based on other languages.
If you look hard enough at Kubernetes - perhaps with a Kafka queue in the mix - it will turn into a big Websphere cluster and wave back.
> My biggest fear is that you are in the minority of developers that can single-handedly operate one
You're right. About 6 months ago the University of Canberra, Australia started it's 'Bachelor of Information Technology in Mainframe Computing' degree program because of the shortage.
It won't last forever...
I'm already hearing the term internally and within job postings -> 'repatriation'... bringing it back from AWS.
Yeah, it’s funny how much groupthink happens in businesses where you’d expect there to be purely rational decision making.
There will be a tipping point where the groupthink will switch back to prioritizing in-house data centers.
I am yet to do any major cloud deployment, most of our customers are very keen on having their data on their computers, not someone else's.
So we did some small stuff on AWS and Azure and that was all about it, in max 5 years time the fad will be on the down path.
This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really high-capacity x86 machine? I know that one difference is that you can hot-swap pretty much any component on a mainframe, but would be very curious to learn more about these beasts.
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus.
Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
The other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc).
An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.
"Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.
I almost forgot about those. I'll have to re-read a couple papers.
I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things).
While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one.
What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way.
They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption.
So basically what I got from that is that the benefits are:
- write once, run forever code
- 100% uptime (not "five nines", literally 100% uptime)
- a reference environment that you can rely on (no runtime change, everything that works today will keep workin in 30 years)
- value added consulting from IBM
- a complete set of tools (runtime, database, primitives, libraries etc)
> - 100% uptime (not "five nines", literally 100% uptime)
There's literally no such thing as 100% uptime. You can add more "nines", meaning that disruptions become exponentially less likely, but that's still very different from "100%".
Oh you've probably not heard of Non-Stop environments for example:
https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonSto...
I remember that in late 90's, NonStop was marketed as having some ludicrous (7?) number of nines of uptime, but not 100%.
I think '100% uptime' is pretty much shorthand for "absolutely no reason to turn the machine off or stop services on them running, ever".
As the above link mentions, everything is redundant. CPUs running in (near) lockstep - voting bad cpus out, redundant storage, power etc etc etc.
I've not seen a better write up on IBM's equivalent systems - but I've not had a reason to play with such big iron. NSKs however were much more affordable, and were used in the late 90s/early 2000s in various telcos etc.
I mean, a meteor could hit the DC. But I guess that would count as a decommissioning event, wouldn’t it? So you’re still right, with a proviso—there’s never any reason to turn the machine off with the expectation of ever turning it back on.
A company I worked for (in São Paulo) had a couple failover setups for its Unisys A-series machine, one with a bank across the street and another with a near twin machine running in its factory in Manaus.
If, for some reason, an event decommissioned all three machines, having our computers back online would be a lesser problem - we would probably be better off learning to hunt and make fire.
Well ... unless you want that exact hardware to move DCs.
Your decommissioning event had me in stitches. I might use that phrase in the future :D
Still are in telco, though many are actively being phased out.
Of course there is, it's an insurance buisiness.
A couple of links here:
https://www.ibm.com/support/knowledgecenter/zosbasics/com.ib...
http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf
Plus one that I seem to be posting on a monthly basis these days:
https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf
Here's one I just found, and will be reading tonight I think:
http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf
Enjoy!
Thank you!
No problem :)
If you're really keen, the base z/OS docs can be found here when you've got a spare couple of decades:
https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pag...
As someone who approached this problem before, the document called "Principles of Operation" IS NOT the correct entry point.
Ha, yeah should probably have mentioned that :)
"As late as 2017, IBM reported 92 of the top 100 banks still used mainframes for their core businesses." https://www.tpr.org/post/how-cobol-still-powers-global-econo...
But why ? Other than inertia and legacy reasons that is. Is there some non partisan resource on this ? A paper or a neutral blog ? Whenever you try to research something like this, you end up on either IBM pages or pages by mainframe enthusiasts. I want to know exactly what you are buying compared to a regular high availability database like setup in the cloud.
I work fairly regularly with our mainframe team, and the general reasoning boils down to massive parallelism and consistency when processing critical data. The number of rows our mainframe can process and write with perfect accuracy is leagues ahead of what similarly priced commodity hardware could achieve. There’s a reason the world still pretty much runs on COBOL/Natural and a hardware design from the 70s, it worked well then and has worked well ever since. Slightly related but I’d describe most of my role being getting data from our micro-services down into the mainframe in a format it understands so we can do ‘something’ with it.
Interesting. So something like google bigquery (or any of the various saas offerings by others) would not be a good match for this problem space ? For massive parallelism, we have GPUs, accelerators, and database products built on these things (admittedly, still somewhat new). By consistency, do you mean tolerance to transient errors and the like ? i.e. Beyond whatever you get with ECC ? Or are you talking about memory consistency models ?
I am just constantly surprised that on the one hand FAANG+ which run so much of the internet economy do not use mainframes, and yet banks do.
The promise of the cloud is that it provides you with "trivially", "infinitely", "cost-efficiently" scalable infrastructure. Banks don't need any of that. They can buy a $100k (?) mainframe, a hundred or so devs, and leave that part of the business on autopilot for 20+ years. Why even bother futzing around in the cloud?
> They can buy a $100k (?) mainframe,
You'll find those on e-Bay. A new one will be significantly more expensive.
Compared to the cost of software engineers a few million every few decades for a mainframe aren't that significant either way.
Also, reaching the same kind of capabilities would require a substantial investment in R&D. IBM has been developing zOS for more than 50 years now.
Getting distributed systems correct is very hard. Mainframe architecture allows one to program and deploy as if you had a giant computer with consistent transactions for the most part. Since banks were the early adopters of computation in large scale it’s clear why most still run on mainframe - the transition to cloud architecture isn’t a simple translation, you need to glue a lot of things together to the same levels of parallelism, and probably sacrifice some consistency that you get “for free” on mainframe architecture.
Cloud is cheaper ($ per cycle) at the expense of human brain power to glue it all together.
In addition to what bendbro suggests, which is actually quite close to reality if oversimplified, banks also have sovereignty concerns with data that may have additional classification and regulatory strings attached. In Australia where I’m based, ‘material’ data such as financial and audit information must be kept in country and its veracity attested to, which is why so many large banks here at least still have big iron in managed data centres for most of their core transactional systems.
Edited to specify which sibling I was referring to.
Banks used to store peoples gold, now they store data.
I don't think any serious bank will consider trusting outsider with their core business.
And mainframes are essentially early on premises clouds.
And today's, cloud offerings still don't do what mainframes did as easily.
(and there are several things that cloud does offer that mainframes can't do)
Outsourcing of financial services processing has been a big business for decades. However it is usually provided to small/medium banks or credit unions by specialized firms or by big banks.
A big bank will usually do its own ledger processing. A good configuration is to process in two pairs of data centers more than 200 miles apart in different electric utility interconnection regions, e.g. Arlington/Fort Worth and Dayton/Springfield. Archival storage of transactions may involve additional centers. https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex#Geographi...
Give me a message queue that guarantees exactly-once-delivery to coordinate a cluster of transaction processing mashines and we can talk about banks using commodity hardware.
> (admittedly, still somewhat new)
That's the whole explanation right there. "Move fast and break things" is not the way that sort of business works.
> But why ?
Recapping what i've written in another comment:
- 100% uptime
- reference, full featured platform that does its job doesn't change much during the years
- write once, run forever approach
- everything on premise
Someone posted in a related thread I was nerding out on tonight that mainframes are excellent when you need to do an enormous amount of I/O with 100% uptime and "exactly once" consistency guarantees. According to that poster, you also don't need to build distributed logic or certain kinds of error handling into your software.
My high level view is that with a mainframe you get a lot of the advantages of the cloud without the hassle of implementing a distributed system / data integrity and consistency over the network (ie something like having the AWS console where you can instantiate services but everything is local. Upgrading the account is physically putting more/swapping resources in the Mainframe). Also there is the cost of migrating, these systems work perfectly reliably since more than 30 years.
AFAIK they have incredible redundancy. Think about RAID-1 but for all components, so one CPU might fail and your software won't even notice that.
It goes a bit deeper than that. I work on POWER which is a sibling to the z platform. They share some common architecture and microarchitectural elements.
To get a sense, have a look at the E980’s Reliability, Availability, and Serviceability Manual. [1] that systems NOT a mainframe - but it borrows ideas from it. Even in that system all levels of it are designed for failure, from Customer replaceable units like fans or power supplies to certain busses have extra lanes or wires for parity or spares depending on the design. Internal to the processor logic there’s a large number of internal checkers that watch the architected state of the machine and notice when something abnormal happens. For several things the processor core will unwind its state and retry an op, and if it proceeds, flag itself as having had a “recoverable” error, which usually triggers a service action. If the system can’t recover it halts either the core or the entire platform, and a dump happens from the service processor which then offloads it to the management console for diagnosis.
Z boxes do all of the above - and more. There’s pictures floating around the office of a system that went through and earthquake, fell over (someone didn’t buy the optional earthquake bracing for the rack) and as it fell, it domino’d over along with several other racks. It still running when the techs went to inspect the data center, though it did turn on its “service me” attention light. ;)
[1] https://www.ibm.com/downloads/cas/2RJYYJML
My favorite advantage is Channel I/O:
https://en.wikipedia.org/wiki/Channel_I/O
That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.
I looked at the author's previous post where he details the process of buying the thing. A fair amount of it goes over my head, I must admit. But that final price tag.... You've gotta really love mainframes. I can't imagine the utility bills for it. The power usage was reduced to 1.7kW. I guess some people buy a sports car and some buy a mainframe. Personally, I'd never be able to drop that kind of money for a hobby. I think emulating a mainframe on a fancy PC would scratch my itch. But to each their own.
If you asked me how much I thought 700kg of computer should cost, I probably would have guessed more than $12k. Is this thing modern? Specs don't seem too impressive but I don't know anything about these things.
In a previous post he said it's 8 years old
I was about to think that was old, but then I remembered that mainframes, as I understand it, can aim for decade+ uptime, so that might actually be reasonably fresh?
Interesting it runs straight off normal 3phase mains - the only mainframe I worked (a bit ) on was a small Unisys one and it had its own 400v supply as part of the set up.
The Z114 can be had with a power option to run off of 380-570 volts DC. See page 6 of "zEnterprise 114 System Overview" https://www-01.ibm.com/support/docview.wss?uid=isg2350bc7c39...
400 V is normal 3 phase mains in about 85 % of the world, but whether consumers have 3 phases available or get 1 or 2 phases varies.
Ah yes I should have been specific I should have said a 400v DC.
The Bulk Power Regulator mentioned in the blog post is essentially an 400V DC supply that powers everything else in the rack (including the blowers with threephase motors and their own DC/AC invertor).
So it is similar except the power supply is built on modern power electronics and is quite small and internal in the rack.
>The Bulk Power Regulator mentioned in the blog post is essentially an 400V DC supply that powers everything else in the rack (including the blowers with threephase motors and their own DC/AC invertor).
Any decent motor these days is a brushless DC motor, so it runs on DC voltage, except that it really doesn't. It gets DC voltage from a power supply (or it rectifies AC voltage to get DC, just like your computer's power supply), and then uses that to drive a motor drive (controller) (a box of electronics that runs the motor). The motor drive generates 3-phase AC power which is actually used to power the windings of the physical motor. The motor includes feedback devices which tell the controller the motor shaft's position so it knows where in the AC cycle the output waveform should be.
There's always a lot of confusion about these motors because of this: they're called "brushless DC" (BLDC), but also 3-phase AC, and both are correct. The key is that the controller is really an integral part of the motor. It's similar to how your CPU probably runs on 1.8VDC, but you don't actually supply it that from your computer's power supply, the motherboard has its own voltage regulator to create that. You would never actually make a separate 1.8V power supply for a CPU, because it wouldn't work very well (high losses over distance); it's much more efficient to have the regulator closely coupled to the CPU.
> Any decent motor these days is a brushless DC motor
Most applications are perfectly fine with asynchronous motors which don't require extra power electronics. The increase in efficiency from IE3 asynchronouos to IE4 synchronous/SRM/hybrid is pretty small and costs a bunch.
Depends on what it is. Even the little fans in desktop computers now are BLDC now, with the controller fitting on a tiny PCB inside the hub of the fan. They're simpler versions, though, with hall-effect feedback rather than an encoder. The power electronics these days are dirt cheap.
> Even the little fans in desktop computers now are BLDC now
Those are at the lower end of the power spectrum of electric motors (which spans approximately 10 orders of magnitude from a few mW to hundreds of MW) and have been BLDCs for a very long time. To see something else in a PC you probably have to go all the way back to the IBM PCs.
> The power electronics these days are dirt cheap.
For a <1 W fan, yes, but when you want to run a 5.5 kW saw it still costs real money and the motor is more expensive.
The reason why I specifically mentioned the blower motors is because the blowers in most mainframes I have seen have use small(-ish) industrial 3phase synchronous motors, not purpose-designed BLDC motors.
I was a former mechanical maintenance tech for an auto manufacturer, so this story feels familiar to connecting a CNC lathe, or a multi-axis mill, but reading this guys ham-fisted accounts made me pretty livid..
>The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.
Since this thing can approach the cost of a rolls-royce,and weighs half as much as a prius, movers and millwrights should have been called in. watching two chubby guys in tennis shoes and no gloves move a 700 kilo server in reverse would have given me a heart attack.
> The various guides did hint towards how the phases are connected to various Bulk Power Regulators (BPR) but nothing was very definite.
that should have been your clue to hire an electrician. there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR.
>I tried to do my best to construct a power cable
Make sure you tell your building supervisor and the fire brigade you "tried your best." there is no "tried your best" section in the electrical code. You either construct a cable to spec, or not.
>Now, I assumed that given there is no neutral it has to be a delta system
Since youre handling voltages that can kill you, perhaps a meter?
>The uncertainty was which phase-pair powered the lowest BPR? I guessed A-B
OR your BPR explodes, your HV breaker trips, and you wipe out power for half the building. Christ on his throne.
> there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR
No, in IBM mainframes, the rack component that takes the mains AC and starts doing something with it is, indeed, called a "bulk power regulator". For example, IBM part number 6186-7040, "Bulk Power Regulator".
> that should have been your clue to hire an electrician
Note in the article he did say "I confirmed the wiring scheme with some friends I have that do the power installations at Dreamhack, and even had them do the wiring as they have all the qualifications needed to do so safely and legally."
I'd love to play with one as well and look forward to the later posts in this series.
There has been a lot of mainframe related posts on HN lately.
You could download the Hercules emulator and either a very old legal copy of MVS, or a not-legal torrent of z/OS.
TL;DR put it where you want it and plug it into a 3-phase outlet.*
*which slightly more complicated in Europe.
Why? It shouldn't be. They use 3-phase in Europe just like in the US, for industrial locations. The differences in residential or lower-voltage standards really have nothing to do with industrial power availability.
The author described some shenanigans he had to go through when wiring the power cord to get the power supplies (which expected voltage that is typical in the US) to run right. In the US it would have been a simple matter of using a power cord in the standard configuration and plugging it into the wall.
Looking closely at the photo of the power supply in the article, it looks like it's quite flexible and can handle different voltages, as long as they're 3-phase. It even lists different amp draws for the different voltage levels. In addition, it looks like it can handle 380-570V DC. With today's switching power supply technology, it's not hard to make them able to accept widely differing input voltages, so this is commonly done so you only need one part to cover the entire worldwide market. It looks to me like you can just plug this thing into any 3-phase power you want really.
As for a "standard configuration" in the US, there really isn't such a thing for 3-phase power. There are a few NEMA standards used in industrial applications, but a lot of times, things like that are hard-wired. This isn't a home appliance.