Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job:
1. Get a user to stop logging in as root.
2. Get all users to stop sharing the same login and password for all servers.
3. Get a user to upgrade their app's dependencies to versions newer than 2010.
4. Get a user to use configuration management rather than scp'ing config files from their laptop to the server.
5. Get a user to bake immutable images w/configuration rather than using configuration management.
6. Get a user to switch from Jenkins to GitHub Actions.
7. Get a user to stop keeping one file with all production secrets in S3, and use a secrets vault instead.
8. Convince a user (and management) you need to buy new servers, because although "we haven't had one go down in years", every one has faulty power supply, hard drive, network card, RAM, etc, and the hardware's so old you can't find spare parts.
9. Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old.
10. Get a user to stop using the aws root account's access keys for their application.
11. Get a user to build their application in a container.
12. Get a user to deploy their application without you.
After you complete each one, you get a glass of scotch. Happy Holidays!
Please don't take this as trivialising your issues. But some potential solutions here might be:
> Get a user to stop logging in as root.
Change root password and take them out of sudoers.
> Get all users to stop sharing the same login and password for all servers.
Switch to SSH certificates. It's going to be an initial moment of pain getting everyone set up with private certificates, but ones it's done there wouldn't be any passwords to share.
If you want to get clether, you can even pull SSH public keys from AWS SSM, or Github, or somewhere elsewhere, so that you don't need to manage these certificates yourself.
Then add to the company IT policy that sharing private certificates is a fireable offence.
> Get a user to use configuration management rather than scp'ing config files from their laptop to the server.
Disable `scp` in the sshd_config for that user
> Get a user to stop keeping one file with all production secrets in S3, and use a secrets vault instead.
AWS has some security tooling here that might help. Though you're not going to get that for free.
> Get a user to stop using the aws root account's access keys for their application.
Just remove the keys. Ideally you'd want them to set up OAuth before hand but even a user access key would be better.
Then add hardware MFA on the root account and notification to Slack/Teams/Email you when someone does try to access it.
> Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old.
AWS Security Hub is free for 30 days, so enable that and then show management the failing items. You're now presenting evidence from an authority (AWS, plus CIS, NIST, whoever if you chose to enable their frameworks too) rather than arguing about what management might perciese as your own personal opinion.
This also has the additional benefit that management can then advertise they're CIS / whatever compliant to their customers.
> Get a user to build their application in a container.
> Get a user to deploy their application without you.
These two would normally be the responsibility of the CI/CD processes rather than developer preference. Do you have the authority and bandwidth to create CI/CD pipelines (be it in Jenkins or Github Actions) for their applications? Is there even a standard template for what these pipelines should look like?
Github Actions left a bad taste in my mouth after having it randomly removed authenticated workers from the pool, after their offline for ~5 days.
This was after setting up a relatively complex PR workflow (always on cheap server starts up very expensive build server with specific hardware) only to have it break randomly after a PR didn't come in for a few days. And no indication that this happens, and no workaround from GitHub.
There are better solutions for CI, GitHub 's is half baked.
Roll 2d6, sum result. Your CI migration target is:
2. migrate secret manager. Roll again
3. cloud build
4. gocd
5. jenkins
6. gitlab
7. github actions
8. bamboo
9. codepipeline
10. buildbot
11. team foundation server
12. migrate version control. Roll again
Not in love with its insistence on recreating the container from scratch every step of the pipeline, among a bundle of other irksome quirks. There are certainly worse choices, though.
Yeah I was thinking of using it for us actually. Connects to everything, lots of plugins, etc. I wonder what the hate is from, they are all pretty bad aren't they ?
Will test forgejo's CI first as we'll use the repo anyway, but if it ain't for me, it's going to be jenkins I assume.
I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list:
1. Patch Microsoft exchange with only a three hour outage window
2. Train a user to use onedrive instead of emailing 50mb files and back and forth
3. Setup eight printers for six users. Deal with 9gb printer drivers.
4. Ask an exec if he would please let you add mfa to their mailbox.
5. Sit there calmly while that exec yells like a wwe wrestler about the ways he plans to ruin you in response
6. Debate the cost of a custom mouse pad for one person across three meetings
7. Deploy any standard windows app that expects everyone be an administrator without making everyone an administrator
8. Deploy an app that expects uac disabled without disabling uac
9. Debug some finance persons 9000 line excel function
A lot of these problems seem pretty solveable, if you're the admin of the machine (or cloud system) and the user isn't.
If you don't want a user to log in as root, disable the root password (or change it to something only you know) and disable root ssh. If you want people to stop sharing the same login and password across all servers, there's several ways to do it but the most straightforward one seems like it would be to enforce the use of a hardware key (yubikey or similar) for login. If people aren't using configuration management software and are leaving machines in an inconsistent state, again there are several options but I'd look into this NixOS project: https://github.com/nix-community/impermanence + some policy of rebooting the machines regularly.
If you don't like how users are making use of AWS resources and secrets, then set up AWS permissions to force them to do so the correct way. In general if someone is using a system in a bad or insecure way, then after alerting them with some lead time, deliberately break their workflow and force them to come to you in order to make progress. If the thing you suggest is actually the correct course of action for your organization, then it will be worthwhile.
We used to run terminal in browser using https://github.com/yudai/gotty and the entire dev team remapped their Ctrl+w to Ctrl+`. We did frontend and backend development with this setup almost for 1.5 years. Muscles memory and till this date, always have the fear if my actual terminal will get closed if I use Ctlr+w :P
It's not clear that you will need an account to see the problems. Logged in with my account and it's exactly the same page. It's not Dec 1st everywhere yet, so they might open up for everyone when they do open them up.
I would like to see and try to solve the scenarios for myself, not to get meaningless internet points. If you look at their front page, you can do that right now. So why do I have to create an account to even see these special advent scenarios?
Time pressures during christmas/holidays mean that the original calendars were becoming too stressful to handle. Seen several calendars switching to 12 consecutive days or 1 every 2 days challenges.
No, Advent is the liturgical season preceding Christmas, beginning the fourth Sunday before Christmas (which is also the Sunday nearest November 30), it is a period of at least three weeks and one day (the shortest period that can start on a Sunday and include four Sundays.)
The 12 days of Christmas start on Christmas and end on January 5, the eve of the Feast of Epiphany.
12-day advent calendars are a fairly recent invention that mirrors the 12-days of Christmas, but has no direct correspondence to anything in any traditional Christian religious calendar (the more common 24-day format is also a modern, but less recent, invention detached from the religious calendar, that simplifies by ignoring the floating start date of advent and always starting on Dec. 1.)
Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job:
After you complete each one, you get a glass of scotch. Happy Holidays!Please don't take this as trivialising your issues. But some potential solutions here might be:
> Get a user to stop logging in as root.
Change root password and take them out of sudoers.
> Get all users to stop sharing the same login and password for all servers.
Switch to SSH certificates. It's going to be an initial moment of pain getting everyone set up with private certificates, but ones it's done there wouldn't be any passwords to share.
If you want to get clether, you can even pull SSH public keys from AWS SSM, or Github, or somewhere elsewhere, so that you don't need to manage these certificates yourself.
Then add to the company IT policy that sharing private certificates is a fireable offence.
> Get a user to use configuration management rather than scp'ing config files from their laptop to the server.
Disable `scp` in the sshd_config for that user
> Get a user to stop keeping one file with all production secrets in S3, and use a secrets vault instead.
AWS has some security tooling here that might help. Though you're not going to get that for free.
> Get a user to stop using the aws root account's access keys for their application.
Just remove the keys. Ideally you'd want them to set up OAuth before hand but even a user access key would be better.
Then add hardware MFA on the root account and notification to Slack/Teams/Email you when someone does try to access it.
> Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old.
AWS Security Hub is free for 30 days, so enable that and then show management the failing items. You're now presenting evidence from an authority (AWS, plus CIS, NIST, whoever if you chose to enable their frameworks too) rather than arguing about what management might perciese as your own personal opinion.
This also has the additional benefit that management can then advertise they're CIS / whatever compliant to their customers.
> Get a user to build their application in a container.
> Get a user to deploy their application without you.
These two would normally be the responsibility of the CI/CD processes rather than developer preference. Do you have the authority and bandwidth to create CI/CD pipelines (be it in Jenkins or Github Actions) for their applications? Is there even a standard template for what these pipelines should look like?
Re: 6. ... Github Actions
Github Actions left a bad taste in my mouth after having it randomly removed authenticated workers from the pool, after their offline for ~5 days.
This was after setting up a relatively complex PR workflow (always on cheap server starts up very expensive build server with specific hardware) only to have it break randomly after a PR didn't come in for a few days. And no indication that this happens, and no workaround from GitHub.
There are better solutions for CI, GitHub 's is half baked.
bugs happen to all of us. whats your better solution - gitlab?
Roll 2d6, sum result. Your CI migration target is:
somehow i am really liking the kind of people that comment in the comment sections of sysadmin posts. i wonder what personality type this is
GitLab pipelines are really good.
Not in love with its insistence on recreating the container from scratch every step of the pipeline, among a bundle of other irksome quirks. There are certainly worse choices, though.
honestly jenkins really isnt that bad
Yeah I was thinking of using it for us actually. Connects to everything, lots of plugins, etc. I wonder what the hate is from, they are all pretty bad aren't they ?
Will test forgejo's CI first as we'll use the repo anyway, but if it ain't for me, it's going to be jenkins I assume.
[dead]
I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list:
1. Patch Microsoft exchange with only a three hour outage window 2. Train a user to use onedrive instead of emailing 50mb files and back and forth 3. Setup eight printers for six users. Deal with 9gb printer drivers. 4. Ask an exec if he would please let you add mfa to their mailbox. 5. Sit there calmly while that exec yells like a wwe wrestler about the ways he plans to ruin you in response 6. Debate the cost of a custom mouse pad for one person across three meetings 7. Deploy any standard windows app that expects everyone be an administrator without making everyone an administrator 8. Deploy an app that expects uac disabled without disabling uac 9. Debug some finance persons 9000 line excel function
That sounds more like Desktop Support than a SysAdmin role. My condolences if that's the job you landed when interviewing for a SysAdmin role
What you describe sounds more like a MS "Modern Workplace" / IT support in a corporate environment.
> … from Jenkins to GitHub Actions.
Oh, good lord why?
Because sysadmim wants to outsource their responsibilities (and job).
You get me the permissions to do half of this stuff, and I'll do whatever you want.
5. and 6. are a matter of taste (trade-offs), the rest is spot on!
I’d be super interested to see solutions to each, just to learn from.
A lot of these problems seem pretty solveable, if you're the admin of the machine (or cloud system) and the user isn't.
If you don't want a user to log in as root, disable the root password (or change it to something only you know) and disable root ssh. If you want people to stop sharing the same login and password across all servers, there's several ways to do it but the most straightforward one seems like it would be to enforce the use of a hardware key (yubikey or similar) for login. If people aren't using configuration management software and are leaving machines in an inconsistent state, again there are several options but I'd look into this NixOS project: https://github.com/nix-community/impermanence + some policy of rebooting the machines regularly.
If you don't like how users are making use of AWS resources and secrets, then set up AWS permissions to force them to do so the correct way. In general if someone is using a system in a bad or insecure way, then after alerting them with some lead time, deliberately break their workflow and force them to come to you in order to make progress. If the thing you suggest is actually the correct course of action for your organization, then it will be worthwhile.
When I get sad and nothing to do in the world, may be hacking into a sad server's problem seems very interesting
We used to run terminal in browser using https://github.com/yudai/gotty and the entire dev team remapped their Ctrl+w to Ctrl+`. We did frontend and backend development with this setup almost for 1.5 years. Muscles memory and till this date, always have the fear if my actual terminal will get closed if I use Ctlr+w :P
hello, creator here, sorry about that. In this case you can click again on the "Open the Server Terminal in a New Window" button
I feel your pain - bites me from time to time, especially in KVM ;)
I wonder if we could get something like that for k8s, docker and other container ecosystem
[flagged]
well advent of code also needs an account
It’s not necessary to see the problems though
It's not clear that you will need an account to see the problems. Logged in with my account and it's exactly the same page. It's not Dec 1st everywhere yet, so they might open up for everyone when they do open them up.
This also has a paid account and a business account.
Checking out how the platform works was two clicks away: home -> give me a server.
I don't know of any other SaaS which gives you a VM with one click without any registration but we do it.
In any case thanks for the feedback, I've put a button on this /advent page for clarity, cheers
how do you want it to work? do you even sysadmin?
I see: a page offering something interesting but vague.
If you tell me more, I might sign up. If I have to create an account first, I'm walking away.
> how do you want it to work?
I would like to see and try to solve the scenarios for myself, not to get meaningless internet points. If you look at their front page, you can do that right now. So why do I have to create an account to even see these special advent scenarios?
> do you even sysadmin?
Yes.
I think the point is "ok, account is free, then what?"
At 5$/m I might give the paid subscription a try.
what's the deal with 12-days advent calendars lately?
Time pressures during christmas/holidays mean that the original calendars were becoming too stressful to handle. Seen several calendars switching to 12 consecutive days or 1 every 2 days challenges.
Advent of Code went from 25 days to 12 days starting this year, as it took too much time each year to come up with 25 unique challenges [1].
[1] https://adventofcode.com/2025/about#faq_num_days
aren't they canonically 12? 12 days of christmas etc
No, Advent is the liturgical season preceding Christmas, beginning the fourth Sunday before Christmas (which is also the Sunday nearest November 30), it is a period of at least three weeks and one day (the shortest period that can start on a Sunday and include four Sundays.)
The 12 days of Christmas start on Christmas and end on January 5, the eve of the Feast of Epiphany.
12-day advent calendars are a fairly recent invention that mirrors the 12-days of Christmas, but has no direct correspondence to anything in any traditional Christian religious calendar (the more common 24-day format is also a modern, but less recent, invention detached from the religious calendar, that simplifies by ignoring the floating start date of advent and always starting on Dec. 1.)
Don't the 12 Days of Christmas start on the 25th though?
Advent calendars track time until Christmas. “12 days of Christmas” are the twelve days after Christmas.
now we need advent of arts,math etc
[dead]
[flagged]
Don't drag me into this.
Do you have notifications set up or something? xD
No, I just occasionally suffer a failure of self-control when I see my almost-namesake in a comment.
Could you elaborate?