The standard UNIX file system layout exists because hard drives were small at the time and files had to be split across several.
The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal that was popular at the time: 46 years ago.
The reason the delete key is broken in like half of all UNIX / Linux systems is because TTY is short for “teletypewriter” — literally a remote type writer banging out text on dead trees. The carriage can go back one character at a time and overwrite an error but it can’t shift printed text on paper.
Windows was developed in an era of CRT displays and delete works with it consistently.
This is why it cracks me up whenever someone talks a out “Linux being the standard”…
Why would you do this to me I'm going to be in a rabbit hole trying to recreate this machine for the next two years. I've been in the hobby long enough that I only get excited about weird things like this keyboard that have historical significance.
> The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal that was popular at the time: 46 years ago.
vi is more than hjkl, and those four non-mnemonic keybindings make up for it by being extremely easy to press for QWERTY users. The vast majority of other commands are perfectly “reasonable” (easy to memorize and not dependent on any antiquated hardware) — append, backward word, change, delete, end of word, find letter, go to (OK, this one is just an arbitrary prefix), and so forth.
Which is why it makes more sense for \r\n to be the *nix line ending - because there is a distinct "carriage return (on same line)" from "line feed (at same carriage position)" operations possible, and combining them together makes carriage return and line feed to put the cursor at the start of a new line.
And why it makes more sense for \n to be the common Windows line ending because it doesn't have that TTY ancestry.
What's even more nuts about Unix 300-layers-of-tty-emulation \n acting like a CRLF is that \r still works just fine as CR. e.g., you can get a nice little clock with:
while true; do echo -n " $(date)\r"; sleep 1; done
And I wish we had a stronger word than unportable for echo. You see that word tossed around on shell scripting resources where the only system it applies to is a 30 year old version of ksh and it only blows up on a full moon when passed a filename with a literal form feed \f, but echo is a practical portability problem.
Until very recently windows terminal couldn’t select a wrapped line without adding \r\n in between… And it still messes up default color brightness making grey text white until something else messes it back.
Until recently "cls" and "Clear-Host" in Windows Terminal used to actually clear everything so you couldn't accidentally scroll back and see junk that you wanted to get rid of.
The Windows Terminal team copied the UNIX terminal "standard" of only clearing the current viewport.
Now, scrolling back "works" in the sense that it shows partial(!) output from previous commands.
Say you have a script that produces 47 screens worth of output, and what you want to see is somewhere in the middle. You can't judge this visually because the scrollbar is inaccurate and there may be other history.
Before you could just 'cls', re-run the script after a change, scroll back, and see only the updated output.
Now, if you do this you'll get 46 screens of superficially identical looking output, with 47 of the output you wanted to see appended to it. This is super confusing.
It has confused me. I've seen other admins be confused as well and copy-paste the wrong thing.
The Microsoft Terminal team debated this and decided to do the wrong thing on purpose and copy the "Linux way" to pander to Linux admins on Windows.
Meanwhile if you go back through the old newsgroups and mailing lists, the same argument was had by Linux people as well, with the same logic I just outlined. They also decided to do the wrong thing. Why? Because they were pandering to UNIX admins coming across to Linux.
Why does UNIX do this? Because of an error in the coding. That is all.
That's why the shiny new Windows Terminal is broken now, to copy an error that was already a copy of an error.
Why does UNIX do this? Because of an error in the coding. That is all.
Are you saying this specifically or metaphorically? Because I suspect that there was a rationale behind that decision.
See also https://man7.org/linux/man-pages/man1/clear.1.html which explicitly states that it clears scrollback iff a terminal has a specified capability. Which windows terminal being of CRT origin should have. I think the issue here is actually an intersection of some terminal nuance and windows team who blindly copied the behavior that some linux guys were used to because this knowledge was buried under their gnome defaults.
Iow, no need to blame unix and linux, blame those who picked one (stupid) mode of many and used that as the only mode of operation.
ncurses' clear defaults to clearing scrollback and you have to use `-x` to not clear it. I would think most distros use it.
Busybox clear does not clear scrollback nor does it provide an option to do it. But for example on Alpine where Busybox clear is the default, the ncurses package can be installed to get the ncurses clear.
You know vim has been around for like 30 years now right? And the Windows thing has nothing to do with CRT displays. Windows was built around IBM compatible PCs which had standardized keyboards. It was the only keyboard layout Windows needed to support.
> The reason the delete key is broken in like half of all UNIX / Linux systems is because TTY is short for “teletypewriter” — literally a remote type writer banging out text on dead trees. The carriage can go back one character at a time and overwrite an error but it can’t shift printed text on paper.
Which Linux system/terminal/editor are you using where insert mode is not the default?
Honestly, the typewriter interface is pretty useful. A program can make text appear on the screen by just writing ASCII (or these days, UTF-8) bytes. The only sad part is the design of escape sequences, but that's more a consequence of chaotic evolution rather than the fact that it's emulating a typewriter (and most of the evolution happened with the early software terminal emulators).
Honest question: How would you design a simple text outputting protocol? Would it be radically different to how it's done today, or would it be the same basic ideas but cleaned up a bit?
I think that text output protocol is the wrong way of thinking about it. It should be a rich data stream capable of advanced features like objective stuff. Think ps, but then you’re able to navigate the output and perform additional functions. I think a subset of html would work for this.
I think the biggest danger would be scope creep, but a terminal shouldn’t just be thought of as text.
Think ps, but then you’re able to navigate the output and perform additional functions
We already have gtk, qt, appkit, win32, web based frontends for almost everything, and none of them produced anything remotely interoperable. The main advantage of just text is that you see its structure right there in a terminal and may reason about it without referring to the documentation of ps result-related set of structures with all the possible bells and whistles embedded into a large xml-like document schema (which is of course full of legacy and references to other specs, because compatibility and standardization).
Unix chose the midpoint between raw unformatted byte input and output (*argv being a raw byte array and stdout being binary or not existing at all and outputs would be configured by whatever was passed into argv) and structured results or a typed-function interface (like COM or CORBA or XML). Either extreme would have been untenable for a cohesive simple operating system, but I do wish Unix had leaned a bit farther toward the typed-interface design. We even had all the ASCII file/group/record/unit control characters.
> How would you design a simple text outputting protocol?
If I were to design terminal control today, I might have a separate file descriptor for a pipe that is used for control communication instead of sending control codes inline with the text. And I would definitely have a way to query the capabilities of the terminal directly, instead of looking it up in a terminfo database.
Having all data in a single steam also makes it easier to buffer messages like a complete log entry with colors and then output that by just writing one buffer to stdout.
Hm, I've actually thought about that a lot. I'd probably keep the bytestream format. It is insanely useful. However I might open up multiple streams. Stream one is basic IO. Stream two could be formatting info. Streams above that implementation dependent, but you could probably start defining them for keyboard/mouse/HID IO or audio and video. Then to make it work over the internet just shove it over SSH or TLS.
The one downside that has is if you separate the formatting from the data you can not recreate the session without keeping time info somehow. So, might need a fix for that.
The only other real alternative I'm aware of is moving the formatting into the API "stream" that communicates using the machine ABI. This is a terrible solution. It's what Windows uses.
There is a thing called the answerback message: computer sends Ctrl-E (0x05, "ENQ"), and the terminal response with the pre-programmed answerback message.
Maybe the message is a shell command, like "rm -rf *" or "chmod a+rwx ."
You can program this message in the setup screen of the terminal. Maybe you can set up all the terminals in the computer center at night...
You can send Ctrl-E to a user with "write" or "wall"..
This answerback message is actually really old, exists even in mechanical ASR-33 teletypes:
I think you can also request the terminal type and at some point you could also request the title of the window in some terminals. Which is why it’s not just annoying but also dangerous to output untrusted characters to the terminal without a filter.
One would hope that we're past the bad times, but at the same time brand new projects like the kitty terminal emulator let stdout read and delete arbitrary files. And verifying something is safe is just hard, these are not even e.g. memory safety bugs, they're "features".
No, kitty does not let stdout out read or delete arbitrary files. First of all stdout is an integer in a lookup table in your kernel, It cant read or delete anything. I assume you mean it lets programs read or delete arbitrary files by writing to and reading from the tty device. And even that is not true. The kitty graphics protocol does let a program delete arbitrary files in the TEMP directory. It does not let anyone either read or list the contents of any part of the filesystem.
Everyone?? No just you. And read the rest of my post. Yes, kitty will delete files in /tmp if asked to. That's what I said. And no unlike your claim, it does not read them. If you cannot tolerate the theoretical possibility of something deleting a file in /tmp ever, then set TEMP before you launch kitty to /tmp/my-mommy-told-me-not-to-put-valuable-things-in-tmp-but-i-didnt-listen
Are you being deliberately obtuse? Yes obviously when asked to display a file, kitty will read it. But it wont send the file contents on the TTY, so the *program sending the request* cannot read it. Which is what we are discussing here. You are trying to make it seem like the ability of a program to get kitty to read an arbitrary file is a security issue. I suggest you go do the following: strace your editor, then ask it to open /etc/passwd, and then be shocked when you see it doing so in the strace log. Seriously, why am I wasting time replying to you.
I'm trying to make it seem like kitty doesn't have very strong security boundaries, and it's a bit of unknown what it can and cannot be made to do.
Me asking my editor to open /etc/passwd is me asking my editor to do something. My terminal emulator opening files and parsing them and deleting them just because I viewed untrusted content is something different. If you want to make that argument, you need to come up with an example where my editor accesses attacker-controlled files because I opened a random README.md.
Ah so classic FUD, "its unknown what it can do". Why dont you spend some effort figuring out what it can actually do instead of making driveby comments with an obvious axe to grind.
You very conveniently left out the fact that pretty much anything that views untrusted content has to parse files. If parsing files is where your "security boundaries" lie, I suggest you drop your computer off at the garbage dump. Hell if your editor has a preview function it too will parse untrusted content. Basically, to do anything software has to parse untrusted content. And just by the way, /etc/passwd is not attacker controlled. If the attacker has access to your filesystem already, she really doesnt need to use kitty to do anything. And "opening" README.md will not cause kitty to parse files, unless by "opening" you mean catting without -v. In which case we are back to your mommy told you to know better but you didn't listen.
At this point its obvious you are deliberately trying to spread FUD. I am done interacting with you. Good bye.
The general trick with parsing is to put it in a sandbox that cannot touch files, can only do limited syscalls, etc. This is what e.g. Chrome does.
But yes, it's obvious the Kitty author has no interest in making it more secure ( https://github.com/kovidgoyal/kitty/issues/2084) and it's obviously you take discussions about improving software security as attacks on your person.
The thing that irritates me is that people then venerate these accidents. Yes, some accidents are happy; many are at best neutral or least-worst or what have you. And then *ix culture dies in a ditch that we must keep all of them, forever, as though they were the result of a carefully thought out process that produced the best of all possible worlds.
UNIX pipes. No more plain text, it has to be typed stream of values. And the terminal will by default print details about it e.g "list of records with this or that format".
Another one that circulated recently was that Qualified Immunity (which means police in the USA can effectively commit civil rights violations without suffering repercussions) partly originated in a clerical error: https://twitter.com/ReichlinMelnick/status/15571788326790144...
"Due to imperfect manufacturing technologies of the time and poor separation between the analog and digital parts of the chip, the 6581's output (before the amplifier stage) was always slightly biased from the zero level. Each time the volume register was altered, an audible click was produced. By quickly adjusting the amplifier's gain through the main 4-bit volume register, this bias could be modulated as PCM, resulting in a "virtual" fourth channel allowing 4-bit digital sample playback. The glitch was known and used from an early point on, first by Electronic Speech Systems to produce sampled speech in games such as Impossible Mission (1983, Epyx) and Ghostbusters (1984, Activision). The first instance of samples being used in actual musical compositions was by Martin Galway in Arkanoid (1987, Imagine), although he had copied the idea from an earlier drum synthesizer package called Digidrums. The length of sampled sound playback was limited first by memory and later technique. Kung Fu Fighting (1986), a popular early sample, has a playback length measured in seconds. c64mp3 (2010) and Cubase64 (2010) demonstrate playback lengths measured in minutes. Also, it was hugely CPU intensive - one had to output the samples very fast (in comparison to the speed of the 6510 CPU)."
>How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?
Not many? In the sense that, considering several UNIX and C design issues and footguns, the dotfiles are the least of our worries (compare e.g. to C buffer overflows or shell escaping rules, to name but two)...
I also don't particularly like the repeated ditches at "lazy programmers" - isn't the whole "New Jersey" style [1] heavy on laziness?
In fact, if those "lazy programmers" were given OS level-APIs and libs doing the right thing about dotfiles, they wouldn't have recoded them on their own in their programs...
> This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
> Here Bob has ignored Pareto Optimality for the context of phone security and convenience because he could enable automatic updates at no (well little) cost to convenience
This is just not true. Software updates break things all the time. Having software updated behind your back is incredibly inconvenient.
So software has bugs, and sometimes those bugs are bad, so we want to do something to mitigate that risk -- the risk that we might have something as serious as RCE in our software.
To reduce the chances of somebody exploiting that hypothetical remote code execution vulnerability we have special code running on our machines that reaches out to download new code and executes it, blindly trusting that every Jo Schmo who's ever written software we were comfortable using once would never take advantage of that position.
The fact that automatic updates sometimes have some benefits right now seems almost accidental when their fundamental operating principle is to open exactly the same sort of backdoor they're trying to protect against.
Experience has shown us that setting up a secure update mechanism is less risky than not updating your applications regularly.
Is this always the case? Nope. But for most cases, updating often against newly exposed threats is worth the vulnerability that auto update introduces.
>This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
Well, part of the idea is to do the "pareto optimal" work not just implementation-wise, but also in checking whether the tradeoffs are pareto optimal (pareto-ception). Thus, avoid "analysis paralysis"
It's not my intention to second guess Ken, but at least in terms of performance I'd say that the current user home directory structure would bet a hit in the buffer cache since Unix got filesystem buffer caches.
In terms of wasted CPU cycles, it's another story but hey these are the days of using npm packages to basically do a string comparison.
Funny. People here are conjecturing why one open codes the comparison instead of calling strcmp() presuming it's C when `ls` was written in assembler where calling a C function like strcmp() is far more onerous (and slow) vs. comparing to an immediate (cmpb, bne).
To be fair, it doesn’t need to be strcmp since the target is known. It could simply check for a null terminator on the next byte or another dot followed by a null terminator. Surely a few more instructions, but not a complex subroutine call.
For example, JSR becomes pretty self explanatory when you know it stands for Jump To Subroutine.
The complexity is more in that you have to do many simple tasks on your own. It may take 50 lines of code in Assembly to do something that Java can do in one line of code.
> The complexity is more in that you have to do many simple tasks on your own.
And, especially while reading it, you have to keep track of a lot of the program state in your mind yourself, and that gets messy when you go beyond small, single-purpose chunks of code.
Pike may be right that it was a coding error (but cf comments elsewhere re the unlikelihood that Ritchie, e.g., wouldn’t have thought of those other cases), but I think he’s wrong about it being a mistake.
I love dotfiles. When I list a folder, I don’t want to see all the cruft, I want to see the key stuff. Sometimes when I organizing something new, I use .directoryName to hide things temporarily.
git is one of my favourite examples: .git hides stuff I do NOT need to see every day, leaving behind the things I am actually working on, the things I care about.
Besides, it is so easy, once one has CLIed a while, to find (no pun intended) and/or account for dot files when you need to.
I still think it is a mistake which you may be repeating by conflating two concepts:
- The location of a file
- Standard visibility of the file
If you hide something (like a folder) temporarily and refer to it through a script, you have to go to the script and change it once you decide that you don't want the folder hidden anymore.
The script is now coupled to not only the location of the resource, but also its visibility.
There is no reason to conflate the two concepts. Git folders can still be hidden by default without requiring adaptation to path names.
I'd argue, what you're loving about dotfiles is not the presence of the dot, but the concept of visibility levels on files.
I understand what you mean, and would argue that the . prefix is a simple - and widely accepted - way of achieving this, as someone noted elsewhere, convention over configuration.
The only way to get truly hidden folders is to agree on an approach and have it adopted everywhere. Someone’s git-aware GUI app might know to hide git, but as a CLI guy, should I have ls modified? Or find? Or my shell (if so, which one)?
The . prefix hides things most users don’t need to see (. and ..) and allows many other things to be usefully hidden in a convenient and consistent manner.
To rephrase my original comment and integrate the crux of yours, Pike is wrong to call this a mistake, because it accidentally gave us a really useful capability that has been widely integrated into most/all of our tools, while allowing us to work around it simply and effectively when we want or need to.
for an alternative that would not have been to hard to implement they could have used the permission bits for hidden files. it would take another bit, but i think there were some spare.
Abusing permission bits would be the same form of problem as abusing names, however permissions are part of a larger bitfield `st_mode` that contains other useful info like file type (e.g. symlink, fifo, etc), which would make sense to extend to include "visible" or whatnot.
Sorry, not trying to knock your idea, just had to let my inner pedant out a bit.
Dotfiles aren't universal either: Windows doesn't hide dotfiles and instead uses a file attribute to mark hidden files. Whatever you do will have portability concerns.
I would never argue that is universal, but just confine myself to the UNIX/Linux world. Were we to argue for universality, we’d be lost within a sea of /, \, -, /, //, \\, drive prefixes, and a few other things.
Within the UNIX/Linux world, though, it’s a remarkably useful and effective convention, implemented and respected with considerable consistency.
(Windows hidden files really aren’t, not once one drops to a CLI, and even then, it depends on the CLI. I’m not offering that as a criticism, but more of a caution of how difficult it is to get something like this to work. Extended attributes and even basic permissions have their twists and difficulties, even within a platform or platform family, let alone cross platform.)
the primary difficulty is to implement a mechanism to carry those attributes in the first place. that's a problem we already have. once that mechanism is there, we can add whatever attributes that we want.
this should have been a concern from the beginning. instead of the hackish workaround of encoding attributes into the filename. they should never have been there. even file-type extensions should have been a separate field of some sort right from the start.
Like they were in Mac OS back in the day, when files had resource forks and data forks?
Or like magic numbers in UNIX/Linux?
The whole extension thing was odd from the beginning. When I first used Windows, after years of UNIX, extensions puzzled me. On UNIX, they told ME something, the OS and apps didn’t need them (/etc/magic did the trick), but here they were on Windows being given semantic value.
Ironically, now that I think about it, I found Windows extensions odd for the same reason that some find hiding dotfiles on UNIX/Linux odd. This makes me chuckle ruefully.
like resource forks. yes. or extended attributes today.
magic numbers are a crude, expensive, and not always working workaround to detect the type of a file based on the first bytes of a data-stream. and there are programs that refuse to load a file if it has an extension that is not supported, even if the content of the file would be just fine. (can't think of an example though, but i definitely remember getting irritated when that happened)
what bothered me about extensions in unix is that they were sometimes arbitrary or wrong. especially on the web, where urls had arbitrary extensions that related to how the page was produced, and not the content that was served (which was almost always html. other extensions only matched for files to be downloaded, but you could never trust it, so in most cases it served no purpose)
But combining these concepts is useful - with a .dot you know based on the name alone if it is hidden or not. If there was a hidden flag then whether or not a file is hidden is itself hidden. E.g. you could have something referring to a file by any name which your file browser / ls does not show and but the file is still there and will be loaded - that's unnecessarily surprising.
Given that we live on a timeline it is useful for sure.
Having a dotfile automatically marked as hidden for example is a useful combination that keeps learned traditions when preferred.
> that's unnecessarily surprising.
I found it unnecessarily surprising that dotfiles werent shown when I got started with unix-like systems. I consider "surprise" to be a weak argument in this case, especially since we do have things that refer to a file that both the file browser and ls do not show (but which are still loaded since opendir and readdir do not care about the dot).
Whether or not a file with a dot at the beginning is shown or not is nothing but arbitrary convention.
I'd like my convention to not care about the file path to decide "does ls or the file browser show it", as that is how I prefer different concepts in general: decoupled.
However, I do accept surprise as an argument against hidden files at all (including dotfiles).
> I'm pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.
That is highly implausible, except as a case of extremely distracted coding.
Someone like Thompson or Ritchie wouldn't consciously write a test for just the leading dot without being fully aware that it matches more than . and .. .
Even if the thought had been "nobody in their right mind will have files starting with dot, so who cares", that would still make it a deliberate requirement being consciously implemented and not a mistake.
By distracted coding I mean that situations like this happen: you intend to write a piece of code which does something specific. The code is motivated by certain happy cases. You start coding it and get it into a state in which it runs but doesn't quite do what was intended. Then you get completely distracted by some interruption. You forget that the code hadn't been completed; you mistake the remembered intent for the completion. You run the code and it carries out the intent on the motivating happy cases, and somehow fool yourself into believing it was done.
Maybe Pike meant that it was a mistake to create that requirement.
I don't hate the $HOME/cfg directory idea, but please call it $HOME/.cfg so I don't have to see it. :)
> Someone like Thompson or Ritchie wouldn't consciously write a test for just the leading dot without being fully aware that it matches more than . and .. .
Why not? Alternatively, who said the decision was 'conscious'? It doesn't take a dumb person to make a dumb mistake. Brilliant people get things wrong all the time, and anyone can be hasty or careless in a moment.
I believe I cover all your remarks with the discussion about distracted coding. If it was a coding mistake, it would have to have been something like that.
It's not a coding mistake where some wrong identifier was used or other typo; coding the proper test requires more instructions. You have to check that there is either a 0 byte after the first '.', or else another '.' byte that is followed by a 0.
Here is how someone could be distracted halfway through and forget to finish it. Suppose the code has a structure like this:
for (;;) {
if (getnext(name) < 0)
break;
/* code added to skip . and .. */
if (name[0] != '.') /* if it doesn't start with a dot, print it */
goto print;
/* This can be continued here with more logic without changing the above line;
But it would already skip . and .. with just the above line. */
if (name[1]) { /* two or more chars */
if (name[1] != '.') /* Dot followed by non-dot: print it */
goto print;
if (name[2] != 0) /* Two dots, followed by non-dot: print it */
goto print;
/* confirmed dot-dot */
}
/* confirmed dot or dot-dot */
continue;
print:
puts(name);
}
And the code does have this structure: it tests for the . character, and branches forward if not equal. Whenever that bne branch is taken, the situation is correct.
So this could be corrected with further refinements by adding more instructions after the bne where the branch is not taken, without changing the prior instructions.
If the behavior was indeed not intended, this would be a case of incomplete code. Incomplete code can happen by distraction. You turn your attention to something else, perhaps for the span of several days, and then don't remember the incomplete state. The brain had visualized the complete state, leaving a false memory of that having been done.
On a related note, one element that is gapingly absent from all historic Unix sources is ... test cases!
Compiler test cases, shell test cases, utility test cases, library function (and system call) test cases ...
I recently learned about this "mistake". I always assumed the dotfile was some design feature by ancient "gurus" of UNIX rather than an accident.
There is something nice and immediate about adding a dot to make it invisible to certain things (ls by default, the file explorers in ubuntu by default, etc.)
A bit like file extensions themselves and the weight they carry in Windows.
It sorts of balances in favour of immediacy of the function over "right way to do things", where the right way might be some kind of meta data for the file, or for it's folder.
Funny thing about hiding, is what does it mean to hide? If you can ask your utility to show hidden things. It sort of means "things I reckon people don't wanna see, or shouldn't touch, unless they are willing to take a step to see it".
A bit like a shibboleth where if you can't figure out how to see it, you are too dangerous/dumb to edit it, or even know about it.
You could instead have security settings for that. Maybe you just don't give the user access to the files at all until they elevate from user to "poweruser". Which would be somewhere between user and superuser. Use chmod to set the permissions, instead of a dot filename format.
> I recently learned about this "mistake". I always assumed the dotfile was some design feature by ancient "gurus" of UNIX rather than an accident.
He said the decision was certainly a mistake, which is an error in judgement, and pretty sure it was an accident, which is an unexpected incident. It is a mistake to conflate the two.
I am not sure whether this was a mistake or accident based on those definitions, it is a close call. But thinking in those terms is certainly good for clarity of thought, so thanks!
> Funny thing about hiding, is what does it mean to hide? If you can ask your utility to show hidden things. It sort of means "things I reckon people don't wanna see, or shouldn't touch, unless they are willing to take a step to see it".
One thing I like about hidden files via the dot prefix is that I feel like it makes clear that the files are hidden only by convention. It's not a security feature, it's just a convenience to help spare you the mental overhead of witnessing the clutter.
I am increasingly exasperated by apps that do not follow the XDG specification for config at this point. Having a ton of dotfiles sitting in the root of the home directory is just a maintenance nightmare on every level.
I've gone so far as to keep my config directory visible -- ~/.config is a symlink to ~/config, which actually holds all my config, conveniently a git repository in itself for easy portability and tracking.
It really is a mess and it would be helpful if we had some sort of database for configuration that was reasonably standardized. Some sort of "registry" if you will :D
Better yet, a flat file structure that you can grep with standard utilities. We can put it somewhere convenient, like in dot files in the users home directory. We can even add them to git with something simple like git add *.*
As tempting as that may be, the ability to enforce ANSI SQL constraints (not null, unique, check, foreign key, etc.) could prevent many types of configuration errors.
If you want to grep everything in a sqlite database, then ".dump" it.
$ echo "create table myconfig(param text, value text);
insert into myconfig values('scrollbar', 'wide');
insert into myconfig values('verbose', 'no');" | sqlite3 myconfig.db
$ sqlite3 myconfig.db '.dump myconfig' | grep verbose
INSERT INTO "myconfig" VALUES('verbose','no');
Yes! Let's also make every config file is its own special snowflake with its own syntax, keywords, and escape characters. That way, it's much harder for software to interact with these files and humans will have to write config templates for each deployment.
Oh and we might as well store these config file in a different place on each distro and move them around once in a while just to keep things interesting. Users can ignore upstream documentation and just strace the process if they wanna know where the files are this year.
You are suggesting good ideas, but we should go further. Let us write configuration files in Turing-complete exotic programming languages that almost nobody knows! This way malicious programs won't be able to parse them or perform automated edits, and many malicious humans too.
Ah I left out a few sentences. My passive aggressiveness is directed at the state of Linux distros today, not at the parent. I agree that his scheme is a lot better than the mess we deal with today.
> Grep is a huge part of my workflow.
You can flatten and grep structured binary formats too.
I honestly feel like you were just matching the tone of my comment so no harm done really. Mine was smarmy, yours passive aggressive. Its like a 90s sitcom in here!
Then you couldn't really use git (or any other kind of VCS) to version your config.
Also, not all config really lends itself to rows in a database (or some other binary registry). For example, commands to run on login. Or really anything where order matters, would be a pain to edit.
Why add just a monstrous database engine that will spend time reading and parsing database schema, parsing SQL queries, building execution plans to read several settings? Let's go further and use Elasticsearch (startup time about 1 minute) for this.
SQLite has identical performance vs parsing ad hoc text files assuming you're reading from flash not RAM. It's also a lot faster if you're loading a few settings from a larger file.
SQLite is inefficient. For example: it parses table definitions and SQL queries, builds execution plans at runtime, although this could be done at compile time.
It probably annoys most people who are accustomed to using their preferred text editor to configure their software, and it probably also annoys all the other users who are accustomed to using typical GUI preference windows accessed through the menus to change their configuration.
Things like dconf and about:config exists in no man's land, where neither generic GUI skills nor generic unixy skills are applicable.
>and it probably also annoys all the other users who are accustomed to using typical GUI preference windows accessed through the menus to change their configuration
How is this a problem with dconf? GNOME Settings for example is typical GUI preferences and works as a front-end to the dconf database.
Last time I tried to use it, "GNOME Settings" (or whatever it was calling then) was woefully incomplete and I had to figure out that dconf-editor existed (which wasn't in the menus.)
Same problem with Firefox; hamburger->settings will take you to a GUI that contains maybe 0.01% of all the settings Firefox actually has. You have to be "in the know" to know that about:config exists. I think the first time most firefox users find out about about:config is when they're angrily searching the web for a solution to their problems, which is basically how I discovered dconf-editor too. You don't find either of these by searching through the GUIs like a normal computer user, nor will you find them by poking around in your dotfiles like a seasoned console jockey.
And, once you have more than a trivial number of changes in about:config, you search for something that will allow you to have sane configuration management, and discover, the 'user.js' file.
Any setting in about:config can be set in user.js and your user.js file can be diffed, grepped, version controlled, edited with your favorite editor, and copied between profiles/machines.
Sadly, Mozilla is moving a lot of config out of about:config/user.js and into multiple random sqlite files, so sane config management is getting harder with each new release.
Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this. The reality is that most people need a graphical, hard to break way of changing their settings to use your software.
Edit: Although I agree that some situations may require a readable text format, like a dotfiles repo. I personally use the `dconf-editor` program to find the right keys and the `dconf load` command to sync them from a regedit-like file format
> Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this.
I don't care. They can use OSes designed for their needs, like macOS and Windows.
Not sure why there is such a persistent idea that "everyone using Linux, including entirely non-technical users who just want an appliance" is actually a desirable or realistic goal.
And the funny thing is that everyone does use Unix. Everyone with a smartphone does as both iOS and Android are derivatives of Unix-OSes (Darwin and Linux). Only Fuchsia will be truly non Unix. If it ever makes it out the door.
However those mobile OSes (or ChromeOS) are of course not what we're talking about when we say we love Unix. The same will happen if Linux ever makes it mainstream. Consumers might love it but we won't. I'm happy for it to stay a niche thing. My niche.
I recently disabled a service from running upon user login on a Windows PC, with a commitment to doing it in a way that can be automated and stored in source control.
I had to check literally more than a dozen locations in the registry (some in different hives) just to see where the thing was actually set to autoload. I then had to create new keys in still another location whose type was simply arbitrary binary data, and the only way to determine what actual value was needed there was to examine the registry before and after disabling and then reenabling the startup profile in the GUI.
The Windows registry is an absolute fucking shitshow, and it's a big part of the reason that the OS is inherently hostile to automation. The inherent invisibility of the registry and the systematic underdocumentation of registry schemas for applications and OS components makes figuring out how to do anything in a repeatable way a fucking quest.
> I personally use the `dconf-editor` program to find the right keys
For GUI applications, config files are often undocumented anyway. But to me having to run an application and see how it messes with its configuration in some database when you click buttons is an insane way to figure out how settings are declared or stored. I would so much rather the configuration format be plaintext and documented than be expected to reverse engineer my apps' friggin' config files!
It might not actually matter. Reboot Windows. Sound starts playing out of your monitor, which you didn't even know had speakers. How do you fix it?
(The answer is not to type "sound" into the search bar. That takes you to the "new" sound control panel, which doesn't have all the old features, including disabling sound devices. So you have to open the control panel, go to Devices & Sound, and then you can search around to disable a device.)
The fundamental difference between Windows and Linux here is that if you write an instructional blog post to fix this problem for Windows, you can put some drive-by malware installer on it and make some money. You could obviously malware up similar Linux documentation, but by the time you get discovered you will have only infected 1 person, simply because it's much less widely used. Unfortunate!
Plenty of users find editing configs in a text editor very friendly. Changing away from that would be LESS friendly to those users. Which users are most important to be friendly to?
I find text configs very friendly up until the point that a package manager blows away my changes. This system harks from an era before fully automated software updates when a full time employed sysadmin was expected to manually oversee every single update. It's far easier to automate incremental changes to a binary database than to non-destructively modify a free form text file.
> I find text configs very friendly up until the point that a package manager blows away my changes.
Neither of these is a complete answer, but
1. Distros like NixOS and GuixSD address this successfully by emitting the config files themselves. They don't statefully edit your configs but instead store them in their own language and then translate them as wholes.
2. You can mitigate much of the pain of this with something like etckeeper, which essentially gives you version control for /etc.
There's no reason you can't have both. KDE for example does a pretty good job of having GUIs to edit config, but that config is stored in human editable text files.
> Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this.
You're responding to complaints that settings done this way are user-hostile, opaque, and require a ton of insider knowledge to even find. So if the only goal is user-friendliness, we're moving away from it, not towards it.
That text file and your editor is two layers of abstraction (at least, because there's parsing involved, e.g. how to i specify timouts? sizes? Can I use "MiB", "s", "ms", etc?).
The thing we want to optimize for is "how easy is to change a setting". Firefox's "about:" seems like a reasonable approach for expert users.
But the abstraction that is the text editor is shared across many programs. You can use your favorite text editor to edit any of the plain-text configs that you need. Firefox’s about: and the way to edit it are specific to Firefox.
Well, yes, but we were imagining a world with a sort of universal "about:", I think?
You wouldn't even need that many basic types to cover about 90%+ of the state space. Of course, there would be bonus points if applications could define their own types in a UI-meaningful way.
Almost all of the things that get written to dconf settings are in some typical GUI preference window in whatever application. Firefox’s about:config can be somewhat controlled through a user.js file.
I stopped caring about this by having home-manager do all this for me. I don't really care where the files are anymore because I change them through Nix, not directly, and I can choose where that is located.
"Having a ton of sheep rounded up in one enclosure is a herding nightmare; why can't they be scattered among the numerous meadows, ravines and gulleys that grace these mountains?"
The latter is exactly how you keep sheep. Keeping them in one enclosure is more of a pain in the arse to cope with than having to spend half a day tromping around with a couple of dogs herding them back into an enclosure, and then sorting out which sheep belongs to who.
I'm not quite sure why this is a good analogy, but it feels like it is.
If I ever find time to play with hacking the kernel my first project is to create an extension that automatically redirects all file system lookups for ~/.* to ~/.config/*
I’ll even forgo having my own hidden files there, I don’t care. It’s so annoying.
So you WANT ~/.cache/ in ~/.config/? It's enough of a nightmare when Electron-based apps litter their folders in ~/.config with service worker caches, that'd make it even worse. It's annoying to need a gigantic .gitignore for all the cache files in ~/.config IMO.
I'm exasperated by apps that hide their configuration files deep within my home directory and I have to use strace to figure out where they are. XDG is mostly useful for people who use unix desktops. It's sort of like a compromise between dotfiles and regedit.exe. Because desktops are huge and complex. If you ask your package manager to install one desktop thing, it installs a hundred things. Non desktop UNIX environments don't have that problem. So I don't think CLI programs like htop shouldn't follow XDG. It's non-standard and not the right tool.
> So I don't think CLI programs like htop shouldn't follow XDG.
I'm confused by your double negative. It looks like you're saying CLI tools should follow XDG, but the tone of your post sounds like the opposite.
If the latter, what are you, some of kind of monster? :) If you don't want a specific XDG config dir, set it to ~. Don't force us all to follow some antiquated convention from days of yore.
> Unfortunately, it is quite possible to get difficult to understand errors when working with Home Manager, such as infinite loops with no clear source reference. You should therefore be comfortable using the Nix language and the various tools in the Nix ecosystem. Reading through the Nix Pills document is a good way to familiarize yourself with them.
Thanks, but I’ll just deal with a bunch of dotfiles. This sounds an awful lot like a “and now you have two problems” situation.
Whatever works for you, but it's not really a fair characterization. It adds one problem and solves another dozen, at least in my case. And I don't really run into these difficult to diagnose issues too often.
It's like driving a car instead of walking is trading one problem for two.
I'm to blame actually, it does a lot more than config management, so I mischaracterized it myself.
Home Manager really doesn’t lead to many problems, and if you encounter any the NixOS community is wonderful. You can set up everything without understanding much of the Nix language by looking at other people’s configs on GitHub.
I've been filing a few bug reports on software that puts its config in dotfiles in ~/. Hopefully we can reach critical mass sooner or later. Because the more projects that follow XDG the more pressure on the projects that have not migrated yet. It makes the Linux desktop better for everyone.
The XDG specification is fairly young. It's a bit younger than a decade. So, I don't expect older packages to update to it.
Also, some applications use their older files if they find it, and re-create under XDG spec when they fail to find that file. KDE did that to great extent and success, for example.
> I've gone so far as to keep my config directory visible -- ~/.config is a symlink to ~/config, which actually holds all my config, conveniently a git repository in itself for easy portability and tracking.
Sometimes I wonder how much defaulting to a directory with a leading dot hurt adoption. It gave me the impression that it was designed for or by people who don't manually edit their config files.
It's not defaulting to a directory, but to a variable: "$XDG_CONFIG_HOME".
Also, if you know how to change config of an application for the last 15-16 years, you also know that you're going to edit a "dot file".
KDE stored everything under ".kde" for a long long time, and stored a great tree under it too. rc files as config files and other KDE related files as app specific files.
In fact, the ".config" folder is just an evolution of ".kde" folder. Spec is written by KDE guys.
Most people seem to interpret this as "don't hide things".
I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Of course, at this point in time no such thing will happen, because it's easier to invent 100 new databases than change some assumptions about our core architecture.
> I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks".
Pike isn't even talking about that, only the typical config files in the home dir that clutter and affect performance of file name ops in the home dir.
> (For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)
I have to agree. I can't think of any situation where hiding really helps. If the reason is clutter, the solution should be a dir.
>I can't think of any situation where hiding really helps.
The irony of dot-files is that they became used for metadata, like .git directories, .htcasses files and so on. So, a solution that fakes metadata by abusing names is now used to indicate metadata at another level of the system. There is clearly something missing in file system design.
I agree, sometimes dot files are used for metadata. But say a typical code repo: lots of dirs (vendored deps, compiled libs, generated code) are in regular dirs but git uses a dot-dir. Is there a strong reason for this I'm not getting? You're not supposed to modify either of them, heck even listing can be a shit-show.
I can see the use-case of "when zipping or sending a dir to a different machine, omit the metadata", but this doesn't hold true for code repos, and is sometimes even the opposite of what you want. It'd probably be better with dir name conventions like "cache", "secrets", "config" or similar.
I think what you're pointing out is an example of a real problem. I'm not sure the problem has a name or is noticed by most people. But something is off. In some cases it might be that common conventions (like dotfiles) aren't really well-designed. But I think the more general issue is that the current set of abstractions (file/directory/symlink) are not sufficient to take good care of practical use cases we have for data these days, not matter how cleverly we use them.
Should .git directory reside in the same directory as your source code? Maybe not. Fossil has an interesting compromise in that regard.
Should compiled artefacts reside in the same directory as your source code? Almost certainly not. This seems like a bad idea from some hacky implementation decades ago.
Should tools be able to distinguish between "real" files and metadata files? Almost certainly yes, except there are different types of "metadata" files, as you pointed out. I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Generally, I think if someone without prejudice examined the common problems we have with data these days and designed a "file" system from scratch, they would end up with something vastly different from our current files/folders paradigm.
> Should .git directory reside in the same directory as your source code? Maybe not.
Indeed, many dependency managers (both go mod and cargo iirc) don't vendor deps into your working dir.
> I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Yeah, the ideal solution may be just that, but I'm not 100% convinced that dirname conventions and tooling can't solve the major hurdles. New file systems would take decades even if they were to take on.. And for VCS and other tools you need to account for many different file systems and can't rely on anything but largest-common-denominator features..
> "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks"
xattr's require a system call for each path you want to look up, and then another system call for each value you actually want to retrieve. Plus all the error handling that goes along with those calls. Oh, and they're all strings, so even more work if you want to have an integer attribute. Also, which quota should the space used by the attributes accrue to? Should we split that into "user" and "system" attributes? Whoops, looks like we need an "ad-hoc" namespace here too... this time with actual security implications!
The mechanism exists, but it's more cumbersome than it's worth. That's certainly the way I've felt about it every time I have to muck around with a system that "mistakenly" decided to use them instead of just a database.
In that light, the dot is not a historical mistake. It's an obvious optimization of a particularly popular use case.
Or if they should be used at all. Distros still default into disabling them.
What is quite interesting, because it's all a matter of user interface. It's a change that doesn't even impact on software compatibility. Any distro that decides that a file is hidden due to an attr can just push that change, and nothing will break.
Inventing new databases doesn't stop the old architecture from working, and everything that depends on it. Changing that architecture will. This is simply why.
He's probably just ignorant of how bad things can get. Hidden files in a home directory is a lot better than what other operating systems came up with (random directories and the dreaded registry). I'd call it a win - the search over a home directory inode is not going to bother anyone either way.
They weren't replacing `strcmp(name, ".")`, they were replacing `strcmp(name, ".") == 0 || strcmp(name, "..") == 0`.
It's somewhat easy to see how someone would make this shortcut when programming in C, but they weren't programming in C, they were programming in assembly (as the article describes). It's so much simpler to only check the first char when programming in assembly, so it's easier to see why the shortcut might've been taken.
Exactly, the logic is that "...", "....", "..*", etc. do not exist and besides users should have the common sense to not name stuff like that. Therefore it is much easier to just use the easier to write (assembler) and much faster solution.
"Much" easier? It only needs 3 or 4 extra instructions to check for the null terminator at position 1 or 2 (0 based), which would be an improvement (only 1 or 2 letter filenames starting with dot would be hidden), and a couple of extra to check for dot at position 1. Given all the other things ls has to do, that's negligible.
Indeed, but I believe, that is not the mindset of a person programming in an extremely constrained environment. Care for correctness is younger than "ls".
For all intents and purposes at the time, it was a hack but it worked, and it worked well.
Care for correctness was much more of a thing, when coding was a rare art and not a copy/paste from stack overflow to finish this week's sprint on time.
There was also a lot less code in general so I would expect every line to be considered in much more detail. Also, 'laziness' tends to be a character flaw that manifests itself everywhere in a person's work, not just in one place. And Richie/Kernighan don't exactly have a bad reputation.
I wonder if the '-a' parameter of ls appeared at the same time. If so it definitely wasn't an oversight.
I cut my teeth writing 8086 assembly code where keeping machine code size down to as few bytes as possible was necessary and that still strikes me as very lazy hack - saving a few bytes at the cost of an infinite number of possible filenames being hidden vs just two?
Pike says the code is in assembler - which means it is written by hand.
In assembler `name[0]=='.'` is simple to write. Fetch what name points to
and compare it to a literal `'.'`.
A function call on the other hand requires pushing registers to the stack,
pushing the arguments to the stack, call the function, and then restoring the registers.
So if you are in a hurry and just want to try something, you will write the equivalent to `name[0]=='.'`.
Also, I mean, if you've grown up in the era of multicore gigahertz, where "gigabytes" is where our opening bid on RAM starts, you just don't understand what programming on this era of machine was like. The difference between a single name[0]=='.' and two calls to a string comparison function could very well have resulted in noticeable differences in performance, in real-world cases.
I like to keep an eye on the costs of what code I'm writing to make sure I don't get too 21st century and let my codebase bloat until it's slow as molasses, but at the same time I happily harvest the benefits of usually not caring whether this function call is inlined or not, or caring about the exact number of cycles an indirect function call takes, etc. It's a delicate balance but worthwhile to cultivate it.
It could have checked for ".\0" and "..\0", is the thing. There was no need to bring the whole of strcmp to bear.
I think hiding files with a path convention is Good, Actually, but I'm comfortable being in the minority on that, and it's possible that if we instead had a hidden flag a la chmod permissions I would prefer that.
I'm willing to bet it's conjecture. Someone probably said "We need to hide . and .." and then realised "hey, if we need to hide these things we'll probably need to hide other things as well - how about we just hide anything that starts with .?"
Dotfiles are useful, not only in the $HOME directory, but in every project that has a .git directory, and many other ways. Maybe it was a design accident, but it has got stuck, and many old greybeards like me are used to it, and find good use of it.
This is reminiscent of John MacCarthy pooping on NIL.
"The unexpected appearance of an interpreter tended to freeze the form of the language, and some of the decisions made rather lightheartedly for the ``Recursive functions ...'' paper later proved unfortunate. These included the COND notation for conditional expressions which leads to an unnecessary depth of parentheses, and the use of the number zero to denote the empty list NIL and the truth value false. Besides encouraging pornographic programming, giving a special interpretation to the address 0 has caused difficulties in all subsequent implementations. " (History of Lisp, 1979)
> my home directory has about a hundred dot files [...]. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.
Nothing against software efficiency, but those hundred loop iterations checking a string aren't going to be the big time-waster of your daily life.
IIRC before that Unix had directories, but it wasn't a rooted DAG. I don't recall the specifics, but I suspect you had to know a directory's inode to get to it if you couldn't reach it from where you were.
It was a general graph. Any directory entry could link to any inode. Standardize ., .. and forbid links to directories other than those and from the directory they're in and you get the classic UNIX file tree.
Multics, the predecessor to Unix, was the first OS to use a hierarchical file system. Before that, file systems were typically flat: a single long list of files.
I don't know how the original Unix was, but presumably flat.
I've never seen anybody use these trashy preset directories as intended. Experienced users ignore them or delete them, using their own directory structure instead. Inexperienced users use one of them (usually Downloads or Desktop) while the others are treated as write-sometimes/read-never; e.g. sometimes they accidentally have files saved to those other directories which may as well be that file disappearing into the void never to be seen again.
These presets only serve to confuse or at best annoy. These directories were well-intentioned originally but proved to be a huge mistake. They need to go away.
(Edit: The above rant applies only to Downloads/Music/Video/etc. I think ~/.config and ~/.local are fine. I have no problem using those as intended, and I've seen plenty of other people using them too.)
Well the Music/Video folders are probably less relevant now with the popularity of streaming. If you aren't using ~/Downloads, then that means you went through the extra effort of re-configuring your web browser for some reason. I don't think many people do that.
Overall I think it would be productive to go back and redesign these old folders, even if it breaks some stuff if only to better reflect the way the new generation of consumers understands tech. There are probably a lot of young people who have never used a PC before, and will be confused coming into them with a background of smartphones and tablets.
Maybe we should eliminate the Desktop folder completely, and only allow adding widgets/app launchers/shortcuts to the desktop rather than files. Using the desktop as a folder has always been a questionable practice since you can't even see it when you have a window open.
At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
> Maybe we should eliminate the Desktop folder completely
Seconded. I'm irrationally annoyed that it gets created and sometimes filled with .desktop files while my DE doesn't even use it.
> At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
You can place files in the "Templates" dir to serve as templates for the "right-click > new document" feature in nautilus and some other file explorers.
They exist on macOS and Windows but I tend to agree that these directories are not particularly useful for users.
Apple software tries to organize files but it's a disaster: iTunes used to download movies to ~/Music/iTunes/iTunes Media/Movies, now TV.app downloads them to ~/Movies/TV/Media/Movies (!), and videos I shoot on my iPhone end up in ~/Pictures.
There has always been a tragedy of the commons with these folders, nobody can agree on where a lot of stuff goes.
Should my game saves go in Documents? What about Documents/My Games/Publisher? What about %APPDATA%/Publisher? Hell there's even a few in ~/My Games. Every new game, figuring out where the save data went is an adventure and it's cluttered up every folder it touches. And now for whatever reason everyone is being pushed to put user data in %APPDATA%, which I don't consistently remember how to get to without that shortcut. So who knows!
I just looked up the save data location for Subnautica and it is (I shit you not): %AppData%\..\LocalLow\Unknown Worlds\Subnautica\Subnautica\SavedGames
LocalLow is just %LocalAppData% for low privillege programs.
The Roaming folder (%appdata%) is for platform independent data,
and the LocalAppdata folder fo platform specific data.
You can find the appdata folder in your userhome "C:Users\YourUserName\AppData", but it's hidden by default if you havent changed it.
I am aware of that, those folders have existed as such since at least Windows XP. But only recently have I noticed consumer software placing fungible user data in there.
IIRC at one point you couldn't even open one of the "AppData" homefolder aliases, Explorer would spit out some error message and you'd have to navigate to the hidden folder or key it in manually.
> Windows users use them, Mac users use them, hell even on a fresh profile in Gnome I use them...
Really, you use all of them? When you go on vacation and take some pictures and video of your family, do you then split those files between ~/Video/Beach2022/ and ~/Pictures/Beach2022?
I've never seen anybody do this, not on Windows, MacOS, or any linux DE. Instead you get something like ~/Desktop/Beach2022 or ~/Downloads/vacations/Beach2022 or generally, a single directory anywhere (often on a USB harddrive, not their home directory at all), but never do those vacation files get split by filetype into ~/Videos and ~/Pictures, that seems like utter madness to me.
I think pictures + videos from vacation are the exception. I definitely use the ~/Movies ~/Pictures ~/Music (or ~/Videos) folders to hold those types of media. I do this on macOS, Linux, and Windows.
Not exactly, but as I poke around those folders on my desktop PC, all of them hold various things. If I didn't have a NAS they'd hold a lot more stuff too.
Browsers default into putting files in ~/Downloads, so that one gets used. DEs default into showing anything under ~/Desktop on your screen, so people disable the desktop file browser.
~/Documents is there because Freedesktop.org somehow thought they were dealing with Windows and tried to port the solution of problems Linux does not have. The other directories aren't used on any OS and only serve to annoy people.
I don't know anyone, other than a few tech-ignorant people who put everything on the desktop, who doesn't use them. Where else would you put stuff like music, pictures, documents, etc? These different types of files usually call for different hierarchies, so you've got to do something.
>These different types of files usually call for different hierarchies
I organize my files by subject, not file type.
My files about space stuff go in a directory called 'space'. Files about the space shuttle go in space/shuttle. This directory contains any sort of file related to the space shuttle, be it picture, video, pdf, epub, or otherwise. I don't see any utility in breaking this directory apart into ~/Pictures/shuttle, ~/Videos/shuttle, ~/Documents/shuttle... Do you really do something like that? How far do you take this "different files, different hierarchies" principle? Do pdfs and docx go in two different hierarchies too? Are pngs and jpegs separated? What's your criteria for deciding when different sorts of files get different hierarchies?
What I do is probably more similar to you than you realize. I'm not categorically placing all images in ~/Pictures, rather I'm using that directory primarily for pictures that I've taken. Each photo taking session gets its own sub-directory with the ISO 8601 date followed by a description of the event. This is extremely convenient because software I use (like raw image developers) automatically knows to look in ~/Pictures and I can access my files right away.
Likewise, I'm not using ~/Documents just for stuff with an .odt or .pdf extension. Rather, any sort of human-readable text based file goes there. I have some markdown files that have embedded images, and of course the .png or .svg files exist alongside them in the relevant directories.
~/Music is probably the best example of why the default XDG approach makes sense. You can place everything in an Artist/Album hierarchy, and most music software is written to automatically pick up and scan these directories to keep their internal libraries in sync. There's no fuss.
I don't limit myself to just the XDG directories when there's not a close fit; i also have a ~/Projects for various things I've made and ~/Programs for statically built programs that can't be installed, and so on.
Basically, I've found that the XDG approach does a pretty good job of splitting up files by purpose, allowing you to compartmentalize by avoiding having too many directories at the top level of the hierarchy.
As an aside, the desktop is a great place as a TODO-list / sticky note. Your stuff sitting there is so present and ugly, you will feel anxiety to get it sorted to a proper place.
I am more likely to do something with those files than if they are hidden away in Downloads (which is like a tempdir), or on windows, some User/Documents subfolder somewhere.
My point is that that sort of person is not organizing their files at all. They're not refusing the Linux/Windows/Mac hierarchy and choosing a different one consciously, they just don't understand that the hierarchy even exists.
A lot of people running more "minimalistic" window managers like i3 or dwm or whatnot don't really have a "desktop" like Windows, macOS, GNOME, KDE, etc. have, and everything tends to just go in ~, or some subdirectory of that.
Downloads is fine, IMO. It is basically a temporary files directory. I'd be fine with changing the name to tmp, but I don't really see a reason to change the default.
Downloads is not a temporary files directory, that's what /tmp is for, I for one am using $HOME/Tmp instead of /tmp because /tmp is for whoever doesn't respect $TMPDIR big problem if you don't have much ram and you want to avoid writing to an sdcard. (assuming /tmp is tmpfs)
Okay, it's persistent storage for potentially arbitrarily large files that probably came from a browser, that you haven't gone out of your way to find another location for. You can go pretty far outside the viable scope of /tmp, but still have the files' presence there be logically temporary.
I find Desktop fine, it serves a distinct purpose; Downloads doesn't seem like the greatest idea, but works well enough considering browsers. Documents/Music/Video/&c., though, I definitely concur doesn't provide any obvious value being more than one directory.
I wish I could! But ~/Desktop and ~/Downloads keep getting re-created by ... something :-/ I tried setting XDG_{DESKTOP,DOWNLOAD}_DIR but it seems to get ignored. Been annoying me for years.
Modify ~/$XDG_DIRNAME_DIR/user-dirs.dirs (usually ~/.config/user-dirs.dirs). You can just point them all to $HOME if you don't want any sub-directories created.
>I wish I could! But ~/Desktop and ~/Downloads keep getting re-created by ... something :-/ I tried setting XDG_{DESKTOP,DOWNLOAD}_DIR but it seems to get ignored. Been annoying me for years.
Perhaps something like:
ln -s /dev/null ~/Desktop
ln -s /dev/null ~/Downloads
etc.
You may indeed prefer an empty canvas you have full control over.
However, creating these folders by default is actually beneficial to most users. Most users (especially the vast majority that transitioned from Windows) expect these folders to exist.
I believe the choice to not create these existed and still exists. Yet either all the distros that made that choice are history or they all made the opposite choice.
I can say the same about other choices. I vastly prefer having a TRRS headphone jack and expandable storage on a phone. These do not appear to be priorities for most people.
The choice not to create these directories basically doesn't exist, because all sorts of software will automatically create them if they don't exist, their creation isn't centrally orchestrated. A while back I was using a minimal i3-based desktop and when I launched transmission-gtk it thought I might want a "Downloads" folder, and would create it whenever it launched, even though it had been configures to download files elsewhere.
That not really true. Most desktop environments will create the directories on session startup but depending on the DE that behavior can be disabled (in xfce for example has a service in the session you can disable).
It is true that applications will create these directories if they don't exist. But you can fix this by setting the XDG user directories[0] to something that isn't in your home directory. I set them all to a subdirectory under `~/.xdg-user-dir/`, but you could just as easily set them to `/tmp`. Nothing has recreated ~/Documents on my system in years.
I was not aware of that behaviour. I agree that should not happen. Or the user should be asked if they want that. This should very much be under the control of the system and the apps the user chooses to manage the system. In a capability-based OS, apps would not be able to misbehave like this, unless granted the capability.
I have long since accepted that it’s not my $home after all, but rather a general purpose application config dumping ground. My actual “home” is now a top level directory used only by me.
I agree, but the OpenBSD did some nice things in regards to ~/Downloads
Using unveil() and pledge(), OpenBSD hides many $HOME and system directories from Firefox and chrome. Thus an errant javascript will only see things in ~/Downloads
It's even worse with dotfiles nowadays than before. We have the cluttered "xdg" dotfile directories, and then a bunch of stuff makes random dotfiles/dotdirs in the home directory, and then .profiles/shellrcs/xinits that are basically required. Every single program seems to think you want a scrollback history, a configuration directory, or some random garbage to remind of you of that one time used it.
Complying with the XDG standard seems about as good as a stringly typed OS like Unix can get.
Part of which is making sure that the defaults the program chooses may be overridden by consistent environment variables.
Programs need some way to persist their own state and configuration, after all.
The fact that XDG is more honored in the breach is tiresome, but this can at least be corrected, and it's well enough put together to be a steady-state approach to wrangling dotfiles.
Windows fixed this by just having %APPDATA%, and to me, its a great idea. No .config, no .local, no .share, no .randomfile, each installed program has its directory in hidden directory, and nobody has to know what each directory is for, and sure, this is what ~/.config is for some, but I also see there's some stuff in ~/.local/share and so on.
To expand on this, when a program gets installed, it should write its binary location in a file, then if you ever erase it, a doesthisexistanymore $Config/dir/*/symlink | remove_entire_configuration_directory should be run via cron daily.
A long time ago I liked listening to original soundtracks for games. Winamp had a plugin that would play .psf (et al) files. There were files that contained the extracted code from games for playing the music. The plugin had a tiny builtin emulator that could run the relevant parts. No graphics or input I suppose.
One soundtrack I really liked was .hack, including the leading dot. Given that "hack" part and the overall theme of the games I assume the initial dot was quite intended. But what the creators could not have foreseen was that their soundtrack was missing from the majority of psf mirrors (but some had it obviously). I guess some part of some stack ignored dot-prefixed files when doing the mirroring.
Anyway, although I'm usually opposed to inband signalling, I am a fan of dot-prefix for hiding. Filenames already have so many special corner cases that you would have to treat them very carefully even without this one. The fact that it is embedded in the name means that any system can roundtrip hiddenness - something that cannot be said for metadata.
I got me a copy of The Unix Programming Environment (1984) by Pike and Kernighan last year. It has plenty of hidden small design gems or anecdotes like this one. For example, why files don't need a \EOF char, it's implicit when you don't read more bytes the you have reached the end of a file...
Unix design was about making things work (1) and simple (2).
EOF was around because many early file systems only recorded the number of allocated blocks per file, so some other mechanism was needed to record the precise length in bytes in cases where padding with spaces was not acceptable.
Exactly - think about a tape with multiple files on it - you want some way of knowing you reached the end of a file without having to seek all the way back to some directory.
I’m on the verge of abandoning the home directory. I’ll put My files elsewhere and leave home to the apps.
I thought XDG would surely solve this problem but it’s painfully clear now that most developers just don’t care. `node_modules` and `snap` are good cases in point. They didn’t even bother hiding them! LOL
I feel consternation about symlinks on Windows. I find them extremely useful, but you never know which program is going to ignore them and which program is going to follow them. They're inconsistent. Plus you can't tell by looking at files if they are symlinks or not because they have the same icon as shortcuts.
I recently managed to crash Visual Studio 2022 because I had a symlink to a drive with a hidden $RECYCLE_BIN on it.
If you copy files in Windows the system will copy the contents of symlinks, however many backup programs will not follow symlinks.
Adding . and .. was a mistake by itself. I never needed them, but always had to write conditions to skip them when traversing the file system.
Also, dotfiles don't work well with masks because `*` doesn't match dot-files, and `.*` match all dotfiles including . and .. which causes many programs to fall into endless recursion (for example, `du -sh .*`).
Everything related to dotfiles in Linux is done wrong.
Why move the folder? Why not just symlink config to .config? There's almost certainly some poorly written tools out there that will blow up on the symlink. Why take that risk?
You're correct, moving the folder can expose a poorly written tool, meaning a tool that isn't following the symlink and isn't using XDG_CONFIG_HOME.
To me, it's an advantage to surface the latent problems, so I can message the authors to ask them about updating. So far, all authors are good with updating.
It turns out the bulk of the issues are in internal scripts that hardcoded ~/.config instead of using XDG_CONFIG_HOME. This is an easy fix, and at the same time we shellcheck those scripts.
From a purely esthetic point of view, if you're using a graphical file manager, the symbolic link might be denoted by an ugly icon. Surely there's a solution to that, but that's the only reason I could find.
I have a directory under ~/config, ~/config/dotfiles, that just has files that are intended to be dot-prefixed and linked in the home directory. That avoids any accidental over-linking; I can just run a script that links ~/config/dotfiles/$X to ~/.$X and I'm done.
I agree that it’s annoying that so much junk accumulates in the root of your home dir.
But any consumer-facing OS will need some sort of mechanism of hiding/protecting configuration files/logs/etc. from deletion by clueless users, like how macOS has started hiding ~/Library by default.
The article points out that a better way to do this would be to designate one particular folder for configs, and then that alone could be hidden if desired. But we have what we have instead because a "convenient" hack was misappropriated.
I find the best way to read a Pike statement is to take it as a challenge to your own strongly held belief. I like .dotfiles behaviour and ls -a does not distress me. But, he makes a case it was not deliberate and probably less beneficial than we think.
So I look at that as a reframing: do I have a good rebuttal? No. I have my own muscle memory and comfort factor, but that's all. How would I think if dotfile hiding didn't happen?
Hiding . & .. is arguably as silly. It's not like they can ever not exist but why hide them?
Maybe a dumb question but why do so many apps rely on dotfiles when /etc exists? I'm not making a judgement, this is an earnest question as even though it's meant for system config files, is there any reason that all app configurations shouldn't live there?
If not in /etc, then at least /usr/etc? I'm sure there is some historical reason but some brief searching and I've not really come up with why dotfiles everywhere are preferable to a centralized configuration directory.
Apps can rely on both /etc (for system defaults) and on ~/.something (for per user defaults).
If we used only /etc, then we would need per user entries there, which would be far more difficult, esp. with network-mounted home folders, especially when those are automounted.
Even given Plan9, which I was aware of (hey, rc was my shell for a while in the early 90s), how many tries did it take to get right in Linux, even with that knowledge and experience? At least two, arguably three, and, arguably, unless namespaces, we didn’t really get there.
(On a somewhat related note, I was very excited that when HP bought Apollo they might integrate the // resource naming into HP-UX. That would have been nice. I say related, because there is a great deal of cool from other systems that died with those systems.)
Dotfiles are a per user application config. Other users of the same system are unlikely to want my idiosyncratic zsh setup. Hence the suggestion for $HOME/cfg in the article.
The second comment there has its merit in UNIX. However, it is moot with namespace in Plan 9. You can have different versions of the same directory structure, per process.
An old way, but also it's still the way to hide a file, on *nix systems.
(Hence the "mistake", and the countless hours mentioned in TFA are still adding up, as people forget them when they shouldn't, or don't exclude them when they should…)
that's because they are all based on unix. or is there any remotely common OS besides windows that is not based on unix?
a few decades ago macOS, amiga and atari were rather common, which all don't follow that convention. i don't know about BeOS/haiku. but if that hides dot-files then it's because they ported a lot of unix tools for commandline use.
i could not find anything about OS/2 but i suspect that it worked like windows.
I didn't mean to imply that Windows is uncommon, but that you have to get pretty damn obscure to find another modern desktop or server OS that doesn't do the dot-equals-hidden thing. Windows is the most common on the desktop, obviously, but if you step outside it you're almost certainly in dot-is-hidden territory.
Windows is still so overwhelmingly common that it's very possible to never have occasion to touch "another" desktop or server OS. Especially if you don't go out of your way to manage servers, and doubly so if you don't live in one of the rich economies.
Windows NT was designed by former DEC employees. DEC had a rather low opinion of Unix and dotfiles definitely would have been one of the lazy hacks they disliked.
The question is why anyone feels files like .gitignore should be hidden by default from ls output - there even seems to be disagreement between versions of git whether git itself ignores such files by default (using git add e.g.), though I don't believe it does, can't check currently.
Was finally able to check on one of my repos and git 2.34.1 most definitely does ignore . files by default (both git status and git add), unless you specify --ignore for git status or --force for git add. But my .gitignore (either for the clone or global core.excludesfile) doesn't specify that . files should be ignored, so why is it doing so?
I can't see how to do a git add to add all new files, including dot files, except those in .gitignore.
Dotfile synchronization is a necessary evil. At work, we have a dotfile synchronization system that contains an infrastructure-level yml file of what to save, what are history files, and what to exclude. Every time a modification is made, it saves a revision. You can have multiple dotfile environments also that completely swaps them out. On the work laptop, I have google drive saving a .dotfiles dir with symlinks into them.
>I was wondering why this post had so many comments. Then I found out it was posted on proggit/HN. The discussions there were hardly interesting, however.
For the curious (was a bit annoying to find since gplus is down now and all links redirect to a different subdomain):
Haha, the first comment is of me throwing shade on HN, oh geez. Now I'm curious to read the old HN thread that I thought was so disappointing: https://news.ycombinator.com/item?id=4331855
>Some things you need to hide from the end user.
De-emphasize - maybe, hide - no. Garbage pile that is average Linux home directory is pretty much result of 'out of sight - out of mind' practice, enabled by dotfiles being hidden. I wish Linux settled on single `~/System/` directory, mirroring `/` structure.
On the other hand, dotfiles convention is useful for meta/"parallel" data, `.git` being prime example - something tighly linked with abitrary file structure.
this post is a good example of how un*x hippies with their fried brains from too much drugs* realize one unit of common sense after aeons of contemplation. just like with generics in go. in fact, go itself is just decades of C users very slowly conceding to the idea of basic hygiene. it took them decades to come up with something which is essentially just what was already available in the 90s (pascal, fortran, ada, basic, later java 1.2 or so which also had green threads), but with stuff done a certain way so the boomers can be like "oh see, we did this minor syntactic change THIS way, checkmate, pascal"), in other words they want to have their say in everything
* im not joking, this is what un*x culture appears to essentially boil down to. boomers were all hippies in the 70s despite presenting themselves as "austere" and "mature" "wise" individuals now, which had the unfortunate side effect of too much drugs. further research is needed to clarify whether brain damage is caused by drugs or un*x
any encoding in file names at all is a mistake. this includes character encodings too, which may sound like a non-sequitur but its not. in fact, files should only have unique identities, not names. names should be metadata. folder hierarchy should just be a user defined data structure that can reference these said "files". anyone who tries developing their own OS without copying extremely idiosyncratic un*x ideas and without being bogged down by the overhead of assembly language or C quickly learns this. an example of youngins discovering this is IPFS (and subsequently implementing it on top of broken un*x)
You didn't get a job at freaking AT&T Bell Labs by being a pot smoking hippie. They were the squarest of square professionals--suit, tie, the whole bit.
Well, I was being a little flippant dismissing it as largely a complaint about things not showing up in the terminal, but it's obviously deeper than that, yes.
I think this overall situation is most acutely a lack of standard. Someone up in the nosebleed section rightly pointed out that there should be some environment variable for a config to be written to, and I agree. I also want to stress that there should be the inclusion of a single variable on top of that, that all programs should use when constructing the path to their configuration. USER_CONFIG_DIR or something equally arguable should, if unset, default to $HOME, and if set, be used as the root to construct whatever structure therein is desired, also able to be changed with an environment variable specific to that program. Be it filename or further subdirectory, first using a single reference that is /not overloaded/, like HOME, but defaults to HOME if unset.
First of all, “A lot of other lazy programmers”. That’s cringe. I don’t care who you are, that’s not a way to talk.
Secondly, I think we have all been there. We introduce a bug in an interface/API but then the users of it start depending on this behaviour and before you know it it’s a feature and it’s impossible to fix without breaking the world.
He's obviously a very smart guy and a pioneer and a much more accomplished person than me (or most of us), but Rob Pike seems to make a habit of arrogantly talking down about other people's intelligence or skill:
Re: Go “The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.”
-- Rob Pike
I agree it's really a cringe way of approaching people. In this (dotfile) case there may have been some self-deprecation in the comment though, so it's frankly hard to get too miffed about it.
I think he very accurately described the problem Go is trying to solve which is how do you make new grads productive?
In my experience at AWS people that graduated didn't understand the languages we used. This leads to a bunch of problems, and makes it's easy to write very bad, unmaintainable code. Lots of needless class hierarchies in Java, or a misunderstanding of `this` in JavaScript. Go solves the problem perfectly.
Never had that experience at Google. New grads and interns always seemed sharp. Might just be the superior quality of education out of U of Waterloo, or the hiring bar at our site, or both. I enjoyed working with them.
With new grads, I generally didn't have to teach coding knowledge. It was more about transmitting wisdom around best practices, estimation, architecture, etc.
If anything, to me Go makes a lot of things harder that could be simpler. Lack of generics means boilerplate. Eschewing test frameworks means boilerplate. The structural typing stuff is exotic (but neat I guess) and not familiar to most new programmers. The actor / CSP type concurrency stuff is also exotic and not familiar to most new programmers. I don't see how it's a language for new programmers.
It also seems very much like a rehash of the concepts they had in Limbo so I actually don't find it honest for him to be claiming it was designed for this purpose (to simplify programming). Instead it feels very much like they had a hammer in search of a nail, an itch they've been scratching since the Plan9 days. And that's fine. I just don't buy this pitch about its market niche and the stuff about the relative intelligence of programmers is gratuitous. I've seen some very smart new grads produce amazing Rust code for example.
Anyway, the idea isn't that everyone already knows Go because it uses familiar features for concurrency, typing, etc.
The idea is that Go is such a simple language, that anyone with solid CS experience can pick it up in a short amount of time and start being productive with it.
Calling someone out (Rob Pike or not) for "A lot of other lazy programmers" is cringe. It's clear and accurate communication, and shouldn't need to be workshopped to avoid offending people determined to be offended.
Good point. Rob Pike has nothing interesting to add here. Just an old white dude making us cringe. It's like come on grandpa, its the 2020s, we've all been there.
I love how so many behaviors we depend on were mistakes or temporary fixes
"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move"
The standard UNIX file system layout exists because hard drives were small at the time and files had to be split across several.
The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal that was popular at the time: 46 years ago.
The reason the delete key is broken in like half of all UNIX / Linux systems is because TTY is short for “teletypewriter” — literally a remote type writer banging out text on dead trees. The carriage can go back one character at a time and overwrite an error but it can’t shift printed text on paper.
Windows was developed in an era of CRT displays and delete works with it consistently.
This is why it cracks me up whenever someone talks a out “Linux being the standard”…
> The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal
The Lear Siegler ADM-3A, actually:
https://en.wikipedia.org/wiki/ADM-3A#Legacy
Why would you do this to me I'm going to be in a rabbit hole trying to recreate this machine for the next two years. I've been in the hobby long enough that I only get excited about weird things like this keyboard that have historical significance.
FabGL can emulate an ADM 3A on an ESP32:
http://www.fabglib.org/group___enumerations_ga67edd9a04814c2...
> The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal that was popular at the time: 46 years ago.
vi is more than hjkl, and those four non-mnemonic keybindings make up for it by being extremely easy to press for QWERTY users. The vast majority of other commands are perfectly “reasonable” (easy to memorize and not dependent on any antiquated hardware) — append, backward word, change, delete, end of word, find letter, go to (OK, this one is just an arbitrary prefix), and so forth.
Which is why it makes more sense for \r\n to be the *nix line ending - because there is a distinct "carriage return (on same line)" from "line feed (at same carriage position)" operations possible, and combining them together makes carriage return and line feed to put the cursor at the start of a new line.
And why it makes more sense for \n to be the common Windows line ending because it doesn't have that TTY ancestry.
What's even more nuts about Unix 300-layers-of-tty-emulation \n acting like a CRLF is that \r still works just fine as CR. e.g., you can get a nice little clock with:
I need to use printf to interpret the escape character:
I like my terminals a little bit cleaner:
It might also work with "echo -e".
(don't forget "-e" for echo)
Depends on your echo. Zsh's interpretes escape sequences by default unless the "BSD_ECHO" option is used, bash's doesn't unless "xpg_echo" is on.
And xpg_echo is on by default for bash on macOS, so for users on macOS it looks like echo interprets escapes by default.
This is one of the reasons why echo is hopelessly unportable.
See https://unix.stackexchange.com/a/65819/55665
Learn something new every day!
And I wish we had a stronger word than unportable for echo. You see that word tossed around on shell scripting resources where the only system it applies to is a 30 year old version of ksh and it only blows up on a full moon when passed a filename with a literal form feed \f, but echo is a practical portability problem.
That would make sense except you have them the wrong way around.
I have them the right way around, it's everyone else who has them the wrong way around.
Sometimes when theory doesn't match reality, it's reality that's wrong.
I don't see anything that says which is which in that comment. Just what makes sense.
Until very recently windows terminal couldn’t select a wrapped line without adding \r\n in between… And it still messes up default color brightness making grey text white until something else messes it back.
Until recently "cls" and "Clear-Host" in Windows Terminal used to actually clear everything so you couldn't accidentally scroll back and see junk that you wanted to get rid of.
The Windows Terminal team copied the UNIX terminal "standard" of only clearing the current viewport.
Now, scrolling back "works" in the sense that it shows partial(!) output from previous commands.
Say you have a script that produces 47 screens worth of output, and what you want to see is somewhere in the middle. You can't judge this visually because the scrollbar is inaccurate and there may be other history.
Before you could just 'cls', re-run the script after a change, scroll back, and see only the updated output.
Now, if you do this you'll get 46 screens of superficially identical looking output, with 47 of the output you wanted to see appended to it. This is super confusing.
It has confused me. I've seen other admins be confused as well and copy-paste the wrong thing.
The Microsoft Terminal team debated this and decided to do the wrong thing on purpose and copy the "Linux way" to pander to Linux admins on Windows.
Meanwhile if you go back through the old newsgroups and mailing lists, the same argument was had by Linux people as well, with the same logic I just outlined. They also decided to do the wrong thing. Why? Because they were pandering to UNIX admins coming across to Linux.
Why does UNIX do this? Because of an error in the coding. That is all.
That's why the shiny new Windows Terminal is broken now, to copy an error that was already a copy of an error.
Why does UNIX do this? Because of an error in the coding. That is all.
Are you saying this specifically or metaphorically? Because I suspect that there was a rationale behind that decision.
See also https://man7.org/linux/man-pages/man1/clear.1.html which explicitly states that it clears scrollback iff a terminal has a specified capability. Which windows terminal being of CRT origin should have. I think the issue here is actually an intersection of some terminal nuance and windows team who blindly copied the behavior that some linux guys were used to because this knowledge was buried under their gnome defaults.
Iow, no need to blame unix and linux, blame those who picked one (stupid) mode of many and used that as the only mode of operation.
Another thing, is how now we tend to get CLI scripts for what used to be nice Visual Studio wizards.
Microsot teams are tainting the Windows developer experience trying to cater to the crowd that has been buying Apple gear for GNU/Linux development.
And the worst part is those of us who resemble your remark just download Git Bash et al because Powershell is too verbose anyway.
Because parsing text is so great.
And that's good
Not from point of view of Windows devs used to what "Developers, Developers, Developers" meant in practice.
ncurses' clear defaults to clearing scrollback and you have to use `-x` to not clear it. I would think most distros use it.
Busybox clear does not clear scrollback nor does it provide an option to do it. But for example on Alpine where Busybox clear is the default, the ncurses package can be installed to get the ncurses clear.
"clear" clears scrollback on my machine. alacritty + zsh on arch linux.
You know vim has been around for like 30 years now right? And the Windows thing has nothing to do with CRT displays. Windows was built around IBM compatible PCs which had standardized keyboards. It was the only keyboard layout Windows needed to support.
> The reason the delete key is broken in like half of all UNIX / Linux systems is because TTY is short for “teletypewriter” — literally a remote type writer banging out text on dead trees. The carriage can go back one character at a time and overwrite an error but it can’t shift printed text on paper.
Which Linux system/terminal/editor are you using where insert mode is not the default?
HTTP "Referer" is the one that makes me sad.
The fact that terminals being an emulation of a teletype comes a close second.
And the origins of /usr/bin a definite third.
Honestly, the typewriter interface is pretty useful. A program can make text appear on the screen by just writing ASCII (or these days, UTF-8) bytes. The only sad part is the design of escape sequences, but that's more a consequence of chaotic evolution rather than the fact that it's emulating a typewriter (and most of the evolution happened with the early software terminal emulators).
Honest question: How would you design a simple text outputting protocol? Would it be radically different to how it's done today, or would it be the same basic ideas but cleaned up a bit?
I think that text output protocol is the wrong way of thinking about it. It should be a rich data stream capable of advanced features like objective stuff. Think ps, but then you’re able to navigate the output and perform additional functions. I think a subset of html would work for this.
I think the biggest danger would be scope creep, but a terminal shouldn’t just be thought of as text.
Think ps, but then you’re able to navigate the output and perform additional functions
We already have gtk, qt, appkit, win32, web based frontends for almost everything, and none of them produced anything remotely interoperable. The main advantage of just text is that you see its structure right there in a terminal and may reason about it without referring to the documentation of ps result-related set of structures with all the possible bells and whistles embedded into a large xml-like document schema (which is of course full of legacy and references to other specs, because compatibility and standardization).
Unix chose the midpoint between raw unformatted byte input and output (*argv being a raw byte array and stdout being binary or not existing at all and outputs would be configured by whatever was passed into argv) and structured results or a typed-function interface (like COM or CORBA or XML). Either extreme would have been untenable for a cohesive simple operating system, but I do wish Unix had leaned a bit farther toward the typed-interface design. We even had all the ASCII file/group/record/unit control characters.
> How would you design a simple text outputting protocol?
If I were to design terminal control today, I might have a separate file descriptor for a pipe that is used for control communication instead of sending control codes inline with the text. And I would definitely have a way to query the capabilities of the terminal directly, instead of looking it up in a terminfo database.
> I might have a separate file descriptor for a pipe that is used for control communication instead of sending control codes inline with the text.
How do you keep that synchronized with the text for things like moving the cursor between words?
Having all data in a single steam also makes it easier to buffer messages like a complete log entry with colors and then output that by just writing one buffer to stdout.
Hm, I've actually thought about that a lot. I'd probably keep the bytestream format. It is insanely useful. However I might open up multiple streams. Stream one is basic IO. Stream two could be formatting info. Streams above that implementation dependent, but you could probably start defining them for keyboard/mouse/HID IO or audio and video. Then to make it work over the internet just shove it over SSH or TLS.
The one downside that has is if you separate the formatting from the data you can not recreate the session without keeping time info somehow. So, might need a fix for that.
The only other real alternative I'm aware of is moving the formatting into the API "stream" that communicates using the machine ABI. This is a terrible solution. It's what Windows uses.
I’ve always been curious though, can VT escape sequences be used to perform user actions? E.g. can you coerce stdout to execute an additional command?
There is a thing called the answerback message: computer sends Ctrl-E (0x05, "ENQ"), and the terminal response with the pre-programmed answerback message. Maybe the message is a shell command, like "rm -rf *" or "chmod a+rwx ."
You can program this message in the setup screen of the terminal. Maybe you can set up all the terminals in the computer center at night...
You can send Ctrl-E to a user with "write" or "wall"..
This answerback message is actually really old, exists even in mechanical ASR-33 teletypes:
https://en.wikipedia.org/wiki/Enquiry_character
It can also be triggered by the mysterious "here is" key on the ADM-3A.
https://dave.cheney.net/2017/08/21/the-here-is-key
I think you can also request the terminal type and at some point you could also request the title of the window in some terminals. Which is why it’s not just annoying but also dangerous to output untrusted characters to the terminal without a filter.
TTY emulators have a long history of being vulnerable to all kinds of nasty things. For example: https://seclists.org/fulldisclosure/2003/Feb/att-341/Termula...
One would hope that we're past the bad times, but at the same time brand new projects like the kitty terminal emulator let stdout read and delete arbitrary files. And verifying something is safe is just hard, these are not even e.g. memory safety bugs, they're "features".
No, kitty does not let stdout out read or delete arbitrary files. First of all stdout is an integer in a lookup table in your kernel, It cant read or delete anything. I assume you mean it lets programs read or delete arbitrary files by writing to and reading from the tty device. And even that is not true. The kitty graphics protocol does let a program delete arbitrary files in the TEMP directory. It does not let anyone either read or list the contents of any part of the filesystem.
> First of all stdout is an integer in a lookup table in your kernel, It cant read or delete anything.
Oh come on, you know what everyone means. "Output with potentially hostile content".
Right now it seems kitty will happily delete e.g. /tmp/.X11-unix/X0. That sounds fun.
https://sw.kovidgoyal.net/kitty/graphics-protocol/#the-trans...
https://github.com/kovidgoyal/kitty/blob/5541e3c2ff441aea319...
Everyone?? No just you. And read the rest of my post. Yes, kitty will delete files in /tmp if asked to. That's what I said. And no unlike your claim, it does not read them. If you cannot tolerate the theoretical possibility of something deleting a file in /tmp ever, then set TEMP before you launch kitty to /tmp/my-mommy-told-me-not-to-put-valuable-things-in-tmp-but-i-didnt-listen
Holy excessive combativeness, Batman!
> And no unlike your claim, it does not read them.
Are you being deliberately obtuse? Yes obviously when asked to display a file, kitty will read it. But it wont send the file contents on the TTY, so the *program sending the request* cannot read it. Which is what we are discussing here. You are trying to make it seem like the ability of a program to get kitty to read an arbitrary file is a security issue. I suggest you go do the following: strace your editor, then ask it to open /etc/passwd, and then be shocked when you see it doing so in the strace log. Seriously, why am I wasting time replying to you.
I'm trying to make it seem like kitty doesn't have very strong security boundaries, and it's a bit of unknown what it can and cannot be made to do.
Me asking my editor to open /etc/passwd is me asking my editor to do something. My terminal emulator opening files and parsing them and deleting them just because I viewed untrusted content is something different. If you want to make that argument, you need to come up with an example where my editor accesses attacker-controlled files because I opened a random README.md.
Also, please read https://news.ycombinator.com/newsguidelines.html, and shove your attitude.
Ah so classic FUD, "its unknown what it can do". Why dont you spend some effort figuring out what it can actually do instead of making driveby comments with an obvious axe to grind.
You very conveniently left out the fact that pretty much anything that views untrusted content has to parse files. If parsing files is where your "security boundaries" lie, I suggest you drop your computer off at the garbage dump. Hell if your editor has a preview function it too will parse untrusted content. Basically, to do anything software has to parse untrusted content. And just by the way, /etc/passwd is not attacker controlled. If the attacker has access to your filesystem already, she really doesnt need to use kitty to do anything. And "opening" README.md will not cause kitty to parse files, unless by "opening" you mean catting without -v. In which case we are back to your mommy told you to know better but you didn't listen.
At this point its obvious you are deliberately trying to spread FUD. I am done interacting with you. Good bye.
Because security is hard nobody should even try?
The general trick with parsing is to put it in a sandbox that cannot touch files, can only do limited syscalls, etc. This is what e.g. Chrome does.
But yes, it's obvious the Kitty author has no interest in making it more secure ( https://github.com/kovidgoyal/kitty/issues/2084) and it's obviously you take discussions about improving software security as attacks on your person.
Have a good day, sir.
Oh man, this is rich :))) Let me just leave this for posterity. kitty has had a socket for remote control for years: https://sw.kovidgoyal.net/kitty/conf/#opt-kitty.allow_remote... (see socket-only). It can allow remote control from specific windows while disallowing it generally: https://sw.kovidgoyal.net/kitty/launch/#cmdoption-launch-all...
Furthermore the next release of kitty has capability based security for remote control with public key crypto to keep the data safe: https://github.com/kovidgoyal/kitty/discussions/5320
I suggest you find a better place to move the goalposts in your attempts to smear and spread FUD.
A teleprompter? Is that a typo? Did you mean a teletype?
Sorry yes teletype.
At least that one saves a byte on every request. The cumulative savings are in the terabytes by now.
It saves 2TB/year, just on Google searches! (4e6 requests/minute)
It's a big world. I'm sure there's someone offsetting this by keeping a 2TB hard drive just filled with 'r's.
> keeping a 2TB hard drive just filled with 'r's.
r-chive.org
That’s to balance out Traveller.
The thing that irritates me is that people then venerate these accidents. Yes, some accidents are happy; many are at best neutral or least-worst or what have you. And then *ix culture dies in a ditch that we must keep all of them, forever, as though they were the result of a carefully thought out process that produced the best of all possible worlds.
Humans are the result of billions of happy accidents. It’s not a bad thing to use them as successful experiments.
What would you change and how, ignoring the cost of re-standardization?
UNIX pipes. No more plain text, it has to be typed stream of values. And the terminal will by default print details about it e.g "list of records with this or that format".
That’s PowerShell. Everything is an object, not text. But a lot of people hate it because it’s different.
Hm, have to check it out again. Last time around installing in Linux wasn't straightforward, plus I hate Snap. But will revisit.
Plan 9.
plan 9 got the same treatment your username did. Problem solved XD
Rephrase. As in, no thought went into the name?
Another one that circulated recently was that Qualified Immunity (which means police in the USA can effectively commit civil rights violations without suffering repercussions) partly originated in a clerical error: https://twitter.com/ReichlinMelnick/status/15571788326790144...
Nothing more permanent than a temporary solution.
"The fix is temporary, unless it works" -- Red Green
Like Quake bunnyhopping and the SID 6581 being able to fake a 4th voice.
Any links about the SID thing? I found Quake bunnyhopping, but “sid 6581 four voices” turns up nothing relevant.
"Due to imperfect manufacturing technologies of the time and poor separation between the analog and digital parts of the chip, the 6581's output (before the amplifier stage) was always slightly biased from the zero level. Each time the volume register was altered, an audible click was produced. By quickly adjusting the amplifier's gain through the main 4-bit volume register, this bias could be modulated as PCM, resulting in a "virtual" fourth channel allowing 4-bit digital sample playback. The glitch was known and used from an early point on, first by Electronic Speech Systems to produce sampled speech in games such as Impossible Mission (1983, Epyx) and Ghostbusters (1984, Activision). The first instance of samples being used in actual musical compositions was by Martin Galway in Arkanoid (1987, Imagine), although he had copied the idea from an earlier drum synthesizer package called Digidrums. The length of sampled sound playback was limited first by memory and later technique. Kung Fu Fighting (1986), a popular early sample, has a playback length measured in seconds. c64mp3 (2010) and Cubase64 (2010) demonstrate playback lengths measured in minutes. Also, it was hugely CPU intensive - one had to output the samples very fast (in comparison to the speed of the 6510 CPU)."
-- from Wikipedia: https://en.wikipedia.org/wiki/MOS_Technology_6581
The Garden of Eden was transitory.
Every omnipotent creator implicitly grasps that initial configuration matters as much as the automaton rules.
Messages from the Unseen World
The Universe is the interior of the Light Cone of the Creation
Science is a Differential Equation. Religion is a Boundary Condition
- Alan Turing
>How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?
Not many? In the sense that, considering several UNIX and C design issues and footguns, the dotfiles are the least of our worries (compare e.g. to C buffer overflows or shell escaping rules, to name but two)...
I also don't particularly like the repeated ditches at "lazy programmers" - isn't the whole "New Jersey" style [1] heavy on laziness?
In fact, if those "lazy programmers" were given OS level-APIs and libs doing the right thing about dotfiles, they wouldn't have recoded them on their own in their programs...
[1] https://en.wikipedia.org/wiki/Worse_is_better#New_Jersey_sty...
Not a fan of worse is better.
I have some rough thoughts on expressing why:
> This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
https://www.paretooptimal.dev/pareto-optimality-and-software...
> Here Bob has ignored Pareto Optimality for the context of phone security and convenience because he could enable automatic updates at no (well little) cost to convenience
This is just not true. Software updates break things all the time. Having software updated behind your back is incredibly inconvenient.
Auto updating does so much for security it seems a no brainer.
Is there a way to improve testing while gaining the benefits of auto updates?
Something about automatic updates feels off:
So software has bugs, and sometimes those bugs are bad, so we want to do something to mitigate that risk -- the risk that we might have something as serious as RCE in our software.
To reduce the chances of somebody exploiting that hypothetical remote code execution vulnerability we have special code running on our machines that reaches out to download new code and executes it, blindly trusting that every Jo Schmo who's ever written software we were comfortable using once would never take advantage of that position.
The fact that automatic updates sometimes have some benefits right now seems almost accidental when their fundamental operating principle is to open exactly the same sort of backdoor they're trying to protect against.
Experience has shown us that setting up a secure update mechanism is less risky than not updating your applications regularly.
Is this always the case? Nope. But for most cases, updating often against newly exposed threats is worth the vulnerability that auto update introduces.
Spoken like someone who never had Windows/Linux auto update break a machine.
Yeah, it’s a real problem, but for the masses, which once compromised are used as further vectors for attack, sensible auto update seems smart
The problem now is how to make it safer
Spoken like someone who was born after Code Red.
The 2001 worm? If so, you are sorely mistaken.
Not everybody has security as first priority. Not to mention the update could introduce security issues itself.
As I replied above, experience has shown us, as a general rule, update mechanisms help more than they hurt.
But like with every solution to a problem, you have to decide what’s best for you, your users, and the internet as a whole. Quite the balancing act.
Yeah, just change minor version. Nothing will break.
- 72h continuous work hours later
Why is everyone looking suicidal?
>Not a fan of worse is better.
That's ironic given your handle on HN :)
>This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
Well, part of the idea is to do the "pareto optimal" work not just implementation-wise, but also in checking whether the tradeoffs are pareto optimal (pareto-ception). Thus, avoid "analysis paralysis"
It's not my intention to second guess Ken, but at least in terms of performance I'd say that the current user home directory structure would bet a hit in the buffer cache since Unix got filesystem buffer caches.
In terms of wasted CPU cycles, it's another story but hey these are the days of using npm packages to basically do a string comparison.
That link is awesome. Thanks.
The code in question: https://github.com/dspinellis/unix-history-repo/commit/389a4...
Funny. People here are conjecturing why one open codes the comparison instead of calling strcmp() presuming it's C when `ls` was written in assembler where calling a C function like strcmp() is far more onerous (and slow) vs. comparing to an immediate (cmpb, bne).
To be fair, it doesn’t need to be strcmp since the target is known. It could simply check for a null terminator on the next byte or another dot followed by a null terminator. Surely a few more instructions, but not a complex subroutine call.
I don’t code in C too much but it seems like it would be pretty simple to check if the next index is a null byte.
And the modern reimplementation: https://github.com/coreutils/coreutils/blob/c7920f2/src/ls.c...
All I have to say looking at that is
Absolutely Disgusting.
Not sure why your comment is grey, because
is an absolutely unnecessary code golf bullshit sitting right there.No wonder we call it code. I can read most programs and get a hunch of what's going on. With assembly like this I'm completely blank.
Assembly is actually pretty simple in that there just aren’t that many commands that are commonly used.
https://en.m.wikipedia.org/wiki/X86_instruction_listings
For example, JSR becomes pretty self explanatory when you know it stands for Jump To Subroutine.
The complexity is more in that you have to do many simple tasks on your own. It may take 50 lines of code in Assembly to do something that Java can do in one line of code.
> The complexity is more in that you have to do many simple tasks on your own.
And, especially while reading it, you have to keep track of a lot of the program state in your mind yourself, and that gets messy when you go beyond small, single-purpose chunks of code.
Assembly is easy enough to write, but a nightmare to try to understand what someone else - or you - previously authored without great comments.
Pike may be right that it was a coding error (but cf comments elsewhere re the unlikelihood that Ritchie, e.g., wouldn’t have thought of those other cases), but I think he’s wrong about it being a mistake.
I love dotfiles. When I list a folder, I don’t want to see all the cruft, I want to see the key stuff. Sometimes when I organizing something new, I use .directoryName to hide things temporarily.
git is one of my favourite examples: .git hides stuff I do NOT need to see every day, leaving behind the things I am actually working on, the things I care about.
Besides, it is so easy, once one has CLIed a while, to find (no pun intended) and/or account for dot files when you need to.
I still think it is a mistake which you may be repeating by conflating two concepts: - The location of a file - Standard visibility of the file
If you hide something (like a folder) temporarily and refer to it through a script, you have to go to the script and change it once you decide that you don't want the folder hidden anymore. The script is now coupled to not only the location of the resource, but also its visibility.
There is no reason to conflate the two concepts. Git folders can still be hidden by default without requiring adaptation to path names.
I'd argue, what you're loving about dotfiles is not the presence of the dot, but the concept of visibility levels on files.
I understand what you mean, and would argue that the . prefix is a simple - and widely accepted - way of achieving this, as someone noted elsewhere, convention over configuration.
The only way to get truly hidden folders is to agree on an approach and have it adopted everywhere. Someone’s git-aware GUI app might know to hide git, but as a CLI guy, should I have ls modified? Or find? Or my shell (if so, which one)?
The . prefix hides things most users don’t need to see (. and ..) and allows many other things to be usefully hidden in a convenient and consistent manner.
To rephrase my original comment and integrate the crux of yours, Pike is wrong to call this a mistake, because it accidentally gave us a really useful capability that has been widely integrated into most/all of our tools, while allowing us to work around it simply and effectively when we want or need to.
for an alternative that would not have been to hard to implement they could have used the permission bits for hidden files. it would take another bit, but i think there were some spare.
Abusing permission bits would be the same form of problem as abusing names, however permissions are part of a larger bitfield `st_mode` that contains other useful info like file type (e.g. symlink, fifo, etc), which would make sense to extend to include "visible" or whatnot.
Sorry, not trying to knock your idea, just had to let my inner pedant out a bit.
Then you have problems with file system portability like you get with permissions.
Dotfiles aren't universal either: Windows doesn't hide dotfiles and instead uses a file attribute to mark hidden files. Whatever you do will have portability concerns.
I would never argue that is universal, but just confine myself to the UNIX/Linux world. Were we to argue for universality, we’d be lost within a sea of /, \, -, /, //, \\, drive prefixes, and a few other things.
Within the UNIX/Linux world, though, it’s a remarkably useful and effective convention, implemented and respected with considerable consistency.
(Windows hidden files really aren’t, not once one drops to a CLI, and even then, it depends on the CLI. I’m not offering that as a criticism, but more of a caution of how difficult it is to get something like this to work. Extended attributes and even basic permissions have their twists and difficulties, even within a platform or platform family, let alone cross platform.)
the primary difficulty is to implement a mechanism to carry those attributes in the first place. that's a problem we already have. once that mechanism is there, we can add whatever attributes that we want.
this should have been a concern from the beginning. instead of the hackish workaround of encoding attributes into the filename. they should never have been there. even file-type extensions should have been a separate field of some sort right from the start.
Like they were in Mac OS back in the day, when files had resource forks and data forks?
Or like magic numbers in UNIX/Linux?
The whole extension thing was odd from the beginning. When I first used Windows, after years of UNIX, extensions puzzled me. On UNIX, they told ME something, the OS and apps didn’t need them (/etc/magic did the trick), but here they were on Windows being given semantic value.
Ironically, now that I think about it, I found Windows extensions odd for the same reason that some find hiding dotfiles on UNIX/Linux odd. This makes me chuckle ruefully.
like resource forks. yes. or extended attributes today.
magic numbers are a crude, expensive, and not always working workaround to detect the type of a file based on the first bytes of a data-stream. and there are programs that refuse to load a file if it has an extension that is not supported, even if the content of the file would be just fine. (can't think of an example though, but i definitely remember getting irritated when that happened)
what bothered me about extensions in unix is that they were sometimes arbitrary or wrong. especially on the web, where urls had arbitrary extensions that related to how the page was produced, and not the content that was served (which was almost always html. other extensions only matched for files to be downloaded, but you could never trust it, so in most cases it served no purpose)
That sounds like a standards problem more than something wrong with the idea itself.
yes, thank you. if i had taken a few more minutes to look at the actual data structures, i would have come to the same conclusion.
today i would probably use extended attributes, since they are a lot more flexible
But combining these concepts is useful - with a .dot you know based on the name alone if it is hidden or not. If there was a hidden flag then whether or not a file is hidden is itself hidden. E.g. you could have something referring to a file by any name which your file browser / ls does not show and but the file is still there and will be loaded - that's unnecessarily surprising.
Given that we live on a timeline it is useful for sure. Having a dotfile automatically marked as hidden for example is a useful combination that keeps learned traditions when preferred.
> that's unnecessarily surprising.
I found it unnecessarily surprising that dotfiles werent shown when I got started with unix-like systems. I consider "surprise" to be a weak argument in this case, especially since we do have things that refer to a file that both the file browser and ls do not show (but which are still loaded since opendir and readdir do not care about the dot).
Whether or not a file with a dot at the beginning is shown or not is nothing but arbitrary convention. I'd like my convention to not care about the file path to decide "does ls or the file browser show it", as that is how I prefer different concepts in general: decoupled.
However, I do accept surprise as an argument against hidden files at all (including dotfiles).
If that wasn't already the behavior, people might do things like prefix with "_" or "__" to sort all the cruft to the top. "__git".
> I'm pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.
That is highly implausible, except as a case of extremely distracted coding.
Someone like Thompson or Ritchie wouldn't consciously write a test for just the leading dot without being fully aware that it matches more than . and .. .
Even if the thought had been "nobody in their right mind will have files starting with dot, so who cares", that would still make it a deliberate requirement being consciously implemented and not a mistake.
By distracted coding I mean that situations like this happen: you intend to write a piece of code which does something specific. The code is motivated by certain happy cases. You start coding it and get it into a state in which it runs but doesn't quite do what was intended. Then you get completely distracted by some interruption. You forget that the code hadn't been completed; you mistake the remembered intent for the completion. You run the code and it carries out the intent on the motivating happy cases, and somehow fool yourself into believing it was done.
Maybe Pike meant that it was a mistake to create that requirement.
I don't hate the $HOME/cfg directory idea, but please call it $HOME/.cfg so I don't have to see it. :)
> Someone like Thompson or Ritchie wouldn't consciously write a test for just the leading dot without being fully aware that it matches more than . and .. .
Why not? Alternatively, who said the decision was 'conscious'? It doesn't take a dumb person to make a dumb mistake. Brilliant people get things wrong all the time, and anyone can be hasty or careless in a moment.
I believe I cover all your remarks with the discussion about distracted coding. If it was a coding mistake, it would have to have been something like that.
It's not a coding mistake where some wrong identifier was used or other typo; coding the proper test requires more instructions. You have to check that there is either a 0 byte after the first '.', or else another '.' byte that is followed by a 0.
Here is how someone could be distracted halfway through and forget to finish it. Suppose the code has a structure like this:
And the code does have this structure: it tests for the . character, and branches forward if not equal. Whenever that bne branch is taken, the situation is correct. So this could be corrected with further refinements by adding more instructions after the bne where the branch is not taken, without changing the prior instructions.If the behavior was indeed not intended, this would be a case of incomplete code. Incomplete code can happen by distraction. You turn your attention to something else, perhaps for the span of several days, and then don't remember the incomplete state. The brain had visualized the complete state, leaving a false memory of that having been done.
On a related note, one element that is gapingly absent from all historic Unix sources is ... test cases!
Compiler test cases, shell test cases, utility test cases, library function (and system call) test cases ...
They are nowhere to be seen.
I recently learned about this "mistake". I always assumed the dotfile was some design feature by ancient "gurus" of UNIX rather than an accident.
There is something nice and immediate about adding a dot to make it invisible to certain things (ls by default, the file explorers in ubuntu by default, etc.)
A bit like file extensions themselves and the weight they carry in Windows.
It sorts of balances in favour of immediacy of the function over "right way to do things", where the right way might be some kind of meta data for the file, or for it's folder.
Funny thing about hiding, is what does it mean to hide? If you can ask your utility to show hidden things. It sort of means "things I reckon people don't wanna see, or shouldn't touch, unless they are willing to take a step to see it".
A bit like a shibboleth where if you can't figure out how to see it, you are too dangerous/dumb to edit it, or even know about it.
You could instead have security settings for that. Maybe you just don't give the user access to the files at all until they elevate from user to "poweruser". Which would be somewhere between user and superuser. Use chmod to set the permissions, instead of a dot filename format.
> I recently learned about this "mistake". I always assumed the dotfile was some design feature by ancient "gurus" of UNIX rather than an accident.
He said the decision was certainly a mistake, which is an error in judgement, and pretty sure it was an accident, which is an unexpected incident. It is a mistake to conflate the two.
I am not sure whether this was a mistake or accident based on those definitions, it is a close call. But thinking in those terms is certainly good for clarity of thought, so thanks!
> Funny thing about hiding, is what does it mean to hide? If you can ask your utility to show hidden things. It sort of means "things I reckon people don't wanna see, or shouldn't touch, unless they are willing to take a step to see it".
One thing I like about hidden files via the dot prefix is that I feel like it makes clear that the files are hidden only by convention. It's not a security feature, it's just a convenience to help spare you the mental overhead of witnessing the clutter.
I am increasingly exasperated by apps that do not follow the XDG specification for config at this point. Having a ton of dotfiles sitting in the root of the home directory is just a maintenance nightmare on every level.
I've gone so far as to keep my config directory visible -- ~/.config is a symlink to ~/config, which actually holds all my config, conveniently a git repository in itself for easy portability and tracking.
It really is a mess and it would be helpful if we had some sort of database for configuration that was reasonably standardized. Some sort of "registry" if you will :D
...but preferably using the SQLite engine, and eschewing well-known proprietary formats.
https://news.ycombinator.com/item?id=32275078
Better yet, a flat file structure that you can grep with standard utilities. We can put it somewhere convenient, like in dot files in the users home directory. We can even add them to git with something simple like git add *.*
As tempting as that may be, the ability to enforce ANSI SQL constraints (not null, unique, check, foreign key, etc.) could prevent many types of configuration errors.
If you want to grep everything in a sqlite database, then ".dump" it.
Good tips, thanks! I love sqlite3.
Yes! Let's also make every config file is its own special snowflake with its own syntax, keywords, and escape characters. That way, it's much harder for software to interact with these files and humans will have to write config templates for each deployment.
Oh and we might as well store these config file in a different place on each distro and move them around once in a while just to keep things interesting. Users can ignore upstream documentation and just strace the process if they wanna know where the files are this year.
You are suggesting good ideas, but we should go further. Let us write configuration files in Turing-complete exotic programming languages that almost nobody knows! This way malicious programs won't be able to parse them or perform automated edits, and many malicious humans too.
No need to get passive aggressive, parent has a legitimate point. Grep is a huge part of my workflow.
Ah I left out a few sentences. My passive aggressiveness is directed at the state of Linux distros today, not at the parent. I agree that his scheme is a lot better than the mess we deal with today.
> Grep is a huge part of my workflow.
You can flatten and grep structured binary formats too.
I honestly feel like you were just matching the tone of my comment so no harm done really. Mine was smarmy, yours passive aggressive. Its like a 90s sitcom in here!
Snowflakes are beautiful.
Better yet, put it in a .rec file for the best of both worlds: https://www.gnu.org/software/recutils/
Then you couldn't really use git (or any other kind of VCS) to version your config.
Also, not all config really lends itself to rows in a database (or some other binary registry). For example, commands to run on login. Or really anything where order matters, would be a pain to edit.
Why add just a monstrous database engine that will spend time reading and parsing database schema, parsing SQL queries, building execution plans to read several settings? Let's go further and use Elasticsearch (startup time about 1 minute) for this.
By these rules, we probably shouldn't link to the C library either.
Or Manticore Search (startup time 2 seconds) :)
Manticore vs Elasticsearch startup time benchmark - https://manticoresearch.com/blog/manticore-alternative-to-el...
SQLite has identical performance vs parsing ad hoc text files assuming you're reading from flash not RAM. It's also a lot faster if you're loading a few settings from a larger file.
SQLite is inefficient. For example: it parses table definitions and SQL queries, builds execution plans at runtime, although this could be done at compile time.
Like I said it makes no difference when your SSD takes 40us to respond to a read. Maybe you have an exceptionally slow CPU?
GNOME has dconf[1], but it's only usable through D-Bus, which probably annoys the anti-systemd people
[1] https://en.wikipedia.org/wiki/Dconf
It probably annoys most people who are accustomed to using their preferred text editor to configure their software, and it probably also annoys all the other users who are accustomed to using typical GUI preference windows accessed through the menus to change their configuration.
Things like dconf and about:config exists in no man's land, where neither generic GUI skills nor generic unixy skills are applicable.
>and it probably also annoys all the other users who are accustomed to using typical GUI preference windows accessed through the menus to change their configuration
How is this a problem with dconf? GNOME Settings for example is typical GUI preferences and works as a front-end to the dconf database.
Last time I tried to use it, "GNOME Settings" (or whatever it was calling then) was woefully incomplete and I had to figure out that dconf-editor existed (which wasn't in the menus.)
Same problem with Firefox; hamburger->settings will take you to a GUI that contains maybe 0.01% of all the settings Firefox actually has. You have to be "in the know" to know that about:config exists. I think the first time most firefox users find out about about:config is when they're angrily searching the web for a solution to their problems, which is basically how I discovered dconf-editor too. You don't find either of these by searching through the GUIs like a normal computer user, nor will you find them by poking around in your dotfiles like a seasoned console jockey.
And, once you have more than a trivial number of changes in about:config, you search for something that will allow you to have sane configuration management, and discover, the 'user.js' file.
Any setting in about:config can be set in user.js and your user.js file can be diffed, grepped, version controlled, edited with your favorite editor, and copied between profiles/machines.
Sadly, Mozilla is moving a lot of config out of about:config/user.js and into multiple random sqlite files, so sane config management is getting harder with each new release.
It's not exactly a feature to have several layers of abstraction, when a text editor could suffice.
Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this. The reality is that most people need a graphical, hard to break way of changing their settings to use your software.
Edit: Although I agree that some situations may require a readable text format, like a dotfiles repo. I personally use the `dconf-editor` program to find the right keys and the `dconf load` command to sync them from a regedit-like file format
> Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this.
I don't care. They can use OSes designed for their needs, like macOS and Windows.
Not sure why there is such a persistent idea that "everyone using Linux, including entirely non-technical users who just want an appliance" is actually a desirable or realistic goal.
Agreed.
And the funny thing is that everyone does use Unix. Everyone with a smartphone does as both iOS and Android are derivatives of Unix-OSes (Darwin and Linux). Only Fuchsia will be truly non Unix. If it ever makes it out the door.
However those mobile OSes (or ChromeOS) are of course not what we're talking about when we say we love Unix. The same will happen if Linux ever makes it mainstream. Consumers might love it but we won't. I'm happy for it to stay a niche thing. My niche.
I recently disabled a service from running upon user login on a Windows PC, with a commitment to doing it in a way that can be automated and stored in source control.
I had to check literally more than a dozen locations in the registry (some in different hives) just to see where the thing was actually set to autoload. I then had to create new keys in still another location whose type was simply arbitrary binary data, and the only way to determine what actual value was needed there was to examine the registry before and after disabling and then reenabling the startup profile in the GUI.
The Windows registry is an absolute fucking shitshow, and it's a big part of the reason that the OS is inherently hostile to automation. The inherent invisibility of the registry and the systematic underdocumentation of registry schemas for applications and OS components makes figuring out how to do anything in a repeatable way a fucking quest.
> I personally use the `dconf-editor` program to find the right keys
For GUI applications, config files are often undocumented anyway. But to me having to run an application and see how it messes with its configuration in some database when you click buttons is an insane way to figure out how settings are declared or stored. I would so much rather the configuration format be plaintext and documented than be expected to reverse engineer my apps' friggin' config files!
It might not actually matter. Reboot Windows. Sound starts playing out of your monitor, which you didn't even know had speakers. How do you fix it?
(The answer is not to type "sound" into the search bar. That takes you to the "new" sound control panel, which doesn't have all the old features, including disabling sound devices. So you have to open the control panel, go to Devices & Sound, and then you can search around to disable a device.)
The fundamental difference between Windows and Linux here is that if you write an instructional blog post to fix this problem for Windows, you can put some drive-by malware installer on it and make some money. You could obviously malware up similar Linux documentation, but by the time you get discovered you will have only infected 1 person, simply because it's much less widely used. Unfortunate!
Plenty of users find editing configs in a text editor very friendly. Changing away from that would be LESS friendly to those users. Which users are most important to be friendly to?
I find text configs very friendly up until the point that a package manager blows away my changes. This system harks from an era before fully automated software updates when a full time employed sysadmin was expected to manually oversee every single update. It's far easier to automate incremental changes to a binary database than to non-destructively modify a free form text file.
> I find text configs very friendly up until the point that a package manager blows away my changes.
Neither of these is a complete answer, but
1. Distros like NixOS and GuixSD address this successfully by emitting the config files themselves. They don't statefully edit your configs but instead store them in their own language and then translate them as wholes.
2. You can mitigate much of the pain of this with something like etckeeper, which essentially gives you version control for /etc.
The problem here are UI designers who adore Apple but cannot afford a Macbook. So they target the Linux desktop.
There's no reason you can't have both. KDE for example does a pretty good job of having GUIs to edit config, but that config is stored in human editable text files.
> Unix will never to achieve true user-friendliness and wide adoption if we treat settings like this.
You're responding to complaints that settings done this way are user-hostile, opaque, and require a ton of insider knowledge to even find. So if the only goal is user-friendliness, we're moving away from it, not towards it.
That text file and your editor is two layers of abstraction (at least, because there's parsing involved, e.g. how to i specify timouts? sizes? Can I use "MiB", "s", "ms", etc?).
The thing we want to optimize for is "how easy is to change a setting". Firefox's "about:" seems like a reasonable approach for expert users.
> how to i specify timouts? sizes? Can I use "MiB", "s", "ms", etc?
These issues neither disappear nor improve when you hide the settings in a mystery binary.
I agree, which is why I mentioned the "about:" thing.
It would be nice to have a base level of types that all programs could agree on.
But the abstraction that is the text editor is shared across many programs. You can use your favorite text editor to edit any of the plain-text configs that you need. Firefox’s about: and the way to edit it are specific to Firefox.
Well, yes, but we were imagining a world with a sort of universal "about:", I think?
You wouldn't even need that many basic types to cover about 90%+ of the state space. Of course, there would be bonus points if applications could define their own types in a UI-meaningful way.
Almost all of the things that get written to dconf settings are in some typical GUI preference window in whatever application. Firefox’s about:config can be somewhat controlled through a user.js file.
IBM AIX has "sort of registry". There are some good things in AIX, but querying configurations is ugly as hell
But it's really nice just being able to alter a regular text file for settings for whatever application you're dealing with.
I stopped caring about this by having home-manager do all this for me. I don't really care where the files are anymore because I change them through Nix, not directly, and I can choose where that is located.
Further, using a tmpfs for root (and by extension ~), means that junk doesn’t accumulate (while Home Manager ensures what should be there, is there).
"Having a ton of sheep rounded up in one enclosure is a herding nightmare; why can't they be scattered among the numerous meadows, ravines and gulleys that grace these mountains?"
The latter is exactly how you keep sheep. Keeping them in one enclosure is more of a pain in the arse to cope with than having to spend half a day tromping around with a couple of dogs herding them back into an enclosure, and then sorting out which sheep belongs to who.
I'm not quite sure why this is a good analogy, but it feels like it is.
Stow is a tool designed to support exactly the workflow you describe: https://www.gnu.org/software/stow/
Been using stow for my dotfiles for many years now and I’m really happy with the workflow: https://github.com/spinningarrow/.files
If I ever find time to play with hacking the kernel my first project is to create an extension that automatically redirects all file system lookups for ~/.* to ~/.config/*
I’ll even forgo having my own hidden files there, I don’t care. It’s so annoying.
Don't forget to add the `.config` exception!
So you WANT ~/.cache/ in ~/.config/? It's enough of a nightmare when Electron-based apps litter their folders in ~/.config with service worker caches, that'd make it even worse. It's annoying to need a gigantic .gitignore for all the cache files in ~/.config IMO.
How about glob symlinks as a filesystem feature? :)
I'm exasperated by apps that hide their configuration files deep within my home directory and I have to use strace to figure out where they are. XDG is mostly useful for people who use unix desktops. It's sort of like a compromise between dotfiles and regedit.exe. Because desktops are huge and complex. If you ask your package manager to install one desktop thing, it installs a hundred things. Non desktop UNIX environments don't have that problem. So I don't think CLI programs like htop shouldn't follow XDG. It's non-standard and not the right tool.
> I'm exasperated by apps that hide their configuration files deep within my home directory and I have to use strace to figure out where they are.
That does sound annoying; if only every application would just use ~/.config/appname then your problem would be solved.
> So I don't think CLI programs like htop shouldn't follow XDG.
I'm confused by your double negative. It looks like you're saying CLI tools should follow XDG, but the tone of your post sounds like the opposite.
If the latter, what are you, some of kind of monster? :) If you don't want a specific XDG config dir, set it to ~. Don't force us all to follow some antiquated convention from days of yore.
You should consider Nix home-manager. It keeps config all in one place and explicit.
> Unfortunately, it is quite possible to get difficult to understand errors when working with Home Manager, such as infinite loops with no clear source reference. You should therefore be comfortable using the Nix language and the various tools in the Nix ecosystem. Reading through the Nix Pills document is a good way to familiarize yourself with them.
Thanks, but I’ll just deal with a bunch of dotfiles. This sounds an awful lot like a “and now you have two problems” situation.
Whatever works for you, but it's not really a fair characterization. It adds one problem and solves another dozen, at least in my case. And I don't really run into these difficult to diagnose issues too often.
It's like driving a car instead of walking is trading one problem for two.
I'm to blame actually, it does a lot more than config management, so I mischaracterized it myself.
Home Manager really doesn’t lead to many problems, and if you encounter any the NixOS community is wonderful. You can set up everything without understanding much of the Nix language by looking at other people’s configs on GitHub.
The new problem is pretty fun though!
info on XDG: [the XDG spec](https://standards.freedesktop.org/basedir-spec/basedir-spec-...)
tools that respect XDG, for fellow JS CLI developers:
- https://github.com/davidtheclark/cosmiconfig Find and load configuration from a package.json property, rc file, or CommonJS module. [Check `searchPaths` to implement XDG spec compliance.](https://github.com/davidtheclark/cosmiconfig/issues/152)
- Sindre's libraries use [`env-paths`](https://github.com/sindresorhus/env-paths#pathsconfig) to get paths compliant with this.
- https://github.com/sindresorhus/conf simple config storing (maybe try [conf-cli](https://github.com/natzcam/conf-cli) to manipulate if needed) the successor to [configstore](https://github.com/sindresorhus/conf#how-is-this-different-f...)
- https://github.com/jonschlinkert/data-store conf like datastore but in the shclinkerverse
I've been filing a few bug reports on software that puts its config in dotfiles in ~/. Hopefully we can reach critical mass sooner or later. Because the more projects that follow XDG the more pressure on the projects that have not migrated yet. It makes the Linux desktop better for everyone.
The XDG specification is fairly young. It's a bit younger than a decade. So, I don't expect older packages to update to it.
Also, some applications use their older files if they find it, and re-create under XDG spec when they fail to find that file. KDE did that to great extent and success, for example.
> I've gone so far as to keep my config directory visible -- ~/.config is a symlink to ~/config, which actually holds all my config, conveniently a git repository in itself for easy portability and tracking.
Sometimes I wonder how much defaulting to a directory with a leading dot hurt adoption. It gave me the impression that it was designed for or by people who don't manually edit their config files.
It's not defaulting to a directory, but to a variable: "$XDG_CONFIG_HOME".
Also, if you know how to change config of an application for the last 15-16 years, you also know that you're going to edit a "dot file".
KDE stored everything under ".kde" for a long long time, and stored a great tree under it too. rc files as config files and other KDE related files as app specific files.
In fact, the ".config" folder is just an evolution of ".kde" folder. Spec is written by KDE guys.
> It's not defaulting to a directory, but to a variable: "$XDG_CONFIG_HOME".
By default I'm referring to the following part of the spec and the fact that systems I've used start in this state:
> If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.
I always (since long before XDG became a thing) used ~/etc, in analogy to /etc. Also ~/bin, ~/var, and ~/tmp.
What a genius idea, I'm gonna write a shell function to "permanently unhide" everything in a directory, I love *nix
That "function" for me is just `alias ls='ls -a'`
Not the same, though there are more tools than ls that use directories.
You can do away with that symlink by setting the XDG environment variables.
I do something similar, managing dotfiles is pretty easy with a git repo
Neat idea with the git part.
Most people seem to interpret this as "don't hide things".
I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Of course, at this point in time no such thing will happen, because it's easier to invent 100 new databases than change some assumptions about our core architecture.
> I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks".
Pike isn't even talking about that, only the typical config files in the home dir that clutter and affect performance of file name ops in the home dir.
> (For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)
I have to agree. I can't think of any situation where hiding really helps. If the reason is clutter, the solution should be a dir.
>I can't think of any situation where hiding really helps.
The irony of dot-files is that they became used for metadata, like .git directories, .htcasses files and so on. So, a solution that fakes metadata by abusing names is now used to indicate metadata at another level of the system. There is clearly something missing in file system design.
I agree, sometimes dot files are used for metadata. But say a typical code repo: lots of dirs (vendored deps, compiled libs, generated code) are in regular dirs but git uses a dot-dir. Is there a strong reason for this I'm not getting? You're not supposed to modify either of them, heck even listing can be a shit-show.
I can see the use-case of "when zipping or sending a dir to a different machine, omit the metadata", but this doesn't hold true for code repos, and is sometimes even the opposite of what you want. It'd probably be better with dir name conventions like "cache", "secrets", "config" or similar.
I think what you're pointing out is an example of a real problem. I'm not sure the problem has a name or is noticed by most people. But something is off. In some cases it might be that common conventions (like dotfiles) aren't really well-designed. But I think the more general issue is that the current set of abstractions (file/directory/symlink) are not sufficient to take good care of practical use cases we have for data these days, not matter how cleverly we use them.
Should .git directory reside in the same directory as your source code? Maybe not. Fossil has an interesting compromise in that regard.
Should compiled artefacts reside in the same directory as your source code? Almost certainly not. This seems like a bad idea from some hacky implementation decades ago.
Should tools be able to distinguish between "real" files and metadata files? Almost certainly yes, except there are different types of "metadata" files, as you pointed out. I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Generally, I think if someone without prejudice examined the common problems we have with data these days and designed a "file" system from scratch, they would end up with something vastly different from our current files/folders paradigm.
> Should .git directory reside in the same directory as your source code? Maybe not.
Indeed, many dependency managers (both go mod and cargo iirc) don't vendor deps into your working dir.
> I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Yeah, the ideal solution may be just that, but I'm not 100% convinced that dirname conventions and tooling can't solve the major hurdles. New file systems would take decades even if they were to take on.. And for VCS and other tools you need to account for many different file systems and can't rely on anything but largest-common-denominator features..
> "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks"
xattr's require a system call for each path you want to look up, and then another system call for each value you actually want to retrieve. Plus all the error handling that goes along with those calls. Oh, and they're all strings, so even more work if you want to have an integer attribute. Also, which quota should the space used by the attributes accrue to? Should we split that into "user" and "system" attributes? Whoops, looks like we need an "ad-hoc" namespace here too... this time with actual security implications!
The mechanism exists, but it's more cumbersome than it's worth. That's certainly the way I've felt about it every time I have to muck around with a system that "mistakenly" decided to use them instead of just a database.
In that light, the dot is not a historical mistake. It's an obvious optimization of a particularly popular use case.
I guess one could mention xattrs but there isn’t a lot of consensus on how they should be used.
Or if they should be used at all. Distros still default into disabling them.
What is quite interesting, because it's all a matter of user interface. It's a change that doesn't even impact on software compatibility. Any distro that decides that a file is hidden due to an attr can just push that change, and nothing will break.
Inventing new databases doesn't stop the old architecture from working, and everything that depends on it. Changing that architecture will. This is simply why.
He's probably just ignorant of how bad things can get. Hidden files in a home directory is a lot better than what other operating systems came up with (random directories and the dreaded registry). I'd call it a win - the search over a home directory inode is not going to bother anyone either way.
I don't understand how skipping an entry by comparing it with strcmp(name, ".") == 0 turned into name[0] == '.'
Is it what really happened (lots of programmers, by mistake, wrote a comparison function which is just plain wrong), or Rob Pike is conjecturing?
They weren't replacing `strcmp(name, ".")`, they were replacing `strcmp(name, ".") == 0 || strcmp(name, "..") == 0`.
It's somewhat easy to see how someone would make this shortcut when programming in C, but they weren't programming in C, they were programming in assembly (as the article describes). It's so much simpler to only check the first char when programming in assembly, so it's easier to see why the shortcut might've been taken.
Exactly, the logic is that "...", "....", "..*", etc. do not exist and besides users should have the common sense to not name stuff like that. Therefore it is much easier to just use the easier to write (assembler) and much faster solution.
"Much" easier? It only needs 3 or 4 extra instructions to check for the null terminator at position 1 or 2 (0 based), which would be an improvement (only 1 or 2 letter filenames starting with dot would be hidden), and a couple of extra to check for dot at position 1. Given all the other things ls has to do, that's negligible.
Indeed, but I believe, that is not the mindset of a person programming in an extremely constrained environment. Care for correctness is younger than "ls".
For all intents and purposes at the time, it was a hack but it worked, and it worked well.
Care for correctness was much more of a thing, when coding was a rare art and not a copy/paste from stack overflow to finish this week's sprint on time.
There was also a lot less code in general so I would expect every line to be considered in much more detail. Also, 'laziness' tends to be a character flaw that manifests itself everywhere in a person's work, not just in one place. And Richie/Kernighan don't exactly have a bad reputation.
I wonder if the '-a' parameter of ls appeared at the same time. If so it definitely wasn't an oversight.
Are we sure there weren't other rules or conventions in place at the time that specified filenames had to start with an alphanumeric character?
I cut my teeth writing 8086 assembly code where keeping machine code size down to as few bytes as possible was necessary and that still strikes me as very lazy hack - saving a few bytes at the cost of an infinite number of possible filenames being hidden vs just two?
Pike says the code is in assembler - which means it is written by hand.
In assembler `name[0]=='.'` is simple to write. Fetch what name points to and compare it to a literal `'.'`.
A function call on the other hand requires pushing registers to the stack, pushing the arguments to the stack, call the function, and then restoring the registers.
So if you are in a hurry and just want to try something, you will write the equivalent to `name[0]=='.'`.
Also, I mean, if you've grown up in the era of multicore gigahertz, where "gigabytes" is where our opening bid on RAM starts, you just don't understand what programming on this era of machine was like. The difference between a single name[0]=='.' and two calls to a string comparison function could very well have resulted in noticeable differences in performance, in real-world cases.
I like to keep an eye on the costs of what code I'm writing to make sure I don't get too 21st century and let my codebase bloat until it's slow as molasses, but at the same time I happily harvest the benefits of usually not caring whether this function call is inlined or not, or caring about the exact number of cycles an indirect function call takes, etc. It's a delicate balance but worthwhile to cultivate it.
ditto! Having coded on PDP-1170s and limited to 32K, fighting with link maps, planning common segments etc... We scratched for bytes.
It could have checked for ".\0" and "..\0", is the thing. There was no need to bring the whole of strcmp to bear.
I think hiding files with a path convention is Good, Actually, but I'm comfortable being in the minority on that, and it's possible that if we instead had a hidden flag a la chmod permissions I would prefer that.
I think dotfiles is the best convention I've seen for hidden files.
With "hidden" being a file attribute (like on MS-DOS), you'd have the risk of a filename collision with a file that you don't know is there.
Read that part again:
> It was in assembler then, but the code in question was equivalent to something like this:
> if (name[0] == '.') continue;
> This statement was a little shorter than what it should have been, which is
> if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0) continue;
There's no strcmp() in the first example.
For those who don't know, it should have been equivalent to:
if (name[0] == '.') {
I'm willing to bet it's conjecture. Someone probably said "We need to hide . and .." and then realised "hey, if we need to hide these things we'll probably need to hide other things as well - how about we just hide anything that starts with .?"
He seems to be saying it was a shortcut, so that the line excludes both "." and ".."
Dotfiles are useful, not only in the $HOME directory, but in every project that has a .git directory, and many other ways. Maybe it was a design accident, but it has got stuck, and many old greybeards like me are used to it, and find good use of it.
This is reminiscent of John MacCarthy pooping on NIL.
"The unexpected appearance of an interpreter tended to freeze the form of the language, and some of the decisions made rather lightheartedly for the ``Recursive functions ...'' paper later proved unfortunate. These included the COND notation for conditional expressions which leads to an unnecessary depth of parentheses, and the use of the number zero to denote the empty list NIL and the truth value false. Besides encouraging pornographic programming, giving a special interpretation to the address 0 has caused difficulties in all subsequent implementations. " (History of Lisp, 1979)
the explanation for pornographic programming are very interesting:
https://stackoverflow.com/questions/8547142/what-did-john-mc...
> 0==() has been the emoticon for pornography since 1958.
> The fact that too many implementation details were leaking at a higher level, i.e. showing up too much
> Code that uses intimate knowledge.
> my home directory has about a hundred dot files [...]. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.
Nothing against software efficiency, but those hundred loop iterations checking a string aren't going to be the big time-waster of your daily life.
> when the file system became hierarchical (it had a very different structure early on).
This leaves me to wonder, what was the structure early on?
IIRC before that Unix had directories, but it wasn't a rooted DAG. I don't recall the specifics, but I suspect you had to know a directory's inode to get to it if you couldn't reach it from where you were.
It was a general graph. Any directory entry could link to any inode. Standardize ., .. and forbid links to directories other than those and from the directory they're in and you get the classic UNIX file tree.
Multics, the predecessor to Unix, was the first OS to use a hierarchical file system. Before that, file systems were typically flat: a single long list of files.
I don't know how the original Unix was, but presumably flat.
Don't worry Rob. Freedesktop.org learned and spammed my home dir with Desktop/Document/Downloads/Music/etc/etc/etc...
I've never seen anybody use these trashy preset directories as intended. Experienced users ignore them or delete them, using their own directory structure instead. Inexperienced users use one of them (usually Downloads or Desktop) while the others are treated as write-sometimes/read-never; e.g. sometimes they accidentally have files saved to those other directories which may as well be that file disappearing into the void never to be seen again.
These presets only serve to confuse or at best annoy. These directories were well-intentioned originally but proved to be a huge mistake. They need to go away.
(Edit: The above rant applies only to Downloads/Music/Video/etc. I think ~/.config and ~/.local are fine. I have no problem using those as intended, and I've seen plenty of other people using them too.)
Well the Music/Video folders are probably less relevant now with the popularity of streaming. If you aren't using ~/Downloads, then that means you went through the extra effort of re-configuring your web browser for some reason. I don't think many people do that.
Overall I think it would be productive to go back and redesign these old folders, even if it breaks some stuff if only to better reflect the way the new generation of consumers understands tech. There are probably a lot of young people who have never used a PC before, and will be confused coming into them with a background of smartphones and tablets.
Maybe we should eliminate the Desktop folder completely, and only allow adding widgets/app launchers/shortcuts to the desktop rather than files. Using the desktop as a folder has always been a questionable practice since you can't even see it when you have a window open.
At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
> Maybe we should eliminate the Desktop folder completely
Seconded. I'm irrationally annoyed that it gets created and sometimes filled with .desktop files while my DE doesn't even use it.
> At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
You can place files in the "Templates" dir to serve as templates for the "right-click > new document" feature in nautilus and some other file explorers.
Windows users use them, Mac users use them, hell even on a fresh profile in Gnome I use them...
It is a little weird to see htem on a server OS install though
They exist on macOS and Windows but I tend to agree that these directories are not particularly useful for users.
Apple software tries to organize files but it's a disaster: iTunes used to download movies to ~/Music/iTunes/iTunes Media/Movies, now TV.app downloads them to ~/Movies/TV/Media/Movies (!), and videos I shoot on my iPhone end up in ~/Pictures.
There has always been a tragedy of the commons with these folders, nobody can agree on where a lot of stuff goes.
Should my game saves go in Documents? What about Documents/My Games/Publisher? What about %APPDATA%/Publisher? Hell there's even a few in ~/My Games. Every new game, figuring out where the save data went is an adventure and it's cluttered up every folder it touches. And now for whatever reason everyone is being pushed to put user data in %APPDATA%, which I don't consistently remember how to get to without that shortcut. So who knows!
I just looked up the save data location for Subnautica and it is (I shit you not): %AppData%\..\LocalLow\Unknown Worlds\Subnautica\Subnautica\SavedGames
LocalLow is just %LocalAppData% for low privillege programs. The Roaming folder (%appdata%) is for platform independent data, and the LocalAppdata folder fo platform specific data. You can find the appdata folder in your userhome "C:Users\YourUserName\AppData", but it's hidden by default if you havent changed it.
Edit: clarified what I mean by the Appdata folder
I am aware of that, those folders have existed as such since at least Windows XP. But only recently have I noticed consumer software placing fungible user data in there.
IIRC at one point you couldn't even open one of the "AppData" homefolder aliases, Explorer would spit out some error message and you'd have to navigate to the hidden folder or key it in manually.
> Windows users use them, Mac users use them, hell even on a fresh profile in Gnome I use them...
Really, you use all of them? When you go on vacation and take some pictures and video of your family, do you then split those files between ~/Video/Beach2022/ and ~/Pictures/Beach2022?
I've never seen anybody do this, not on Windows, MacOS, or any linux DE. Instead you get something like ~/Desktop/Beach2022 or ~/Downloads/vacations/Beach2022 or generally, a single directory anywhere (often on a USB harddrive, not their home directory at all), but never do those vacation files get split by filetype into ~/Videos and ~/Pictures, that seems like utter madness to me.
I think pictures + videos from vacation are the exception. I definitely use the ~/Movies ~/Pictures ~/Music (or ~/Videos) folders to hold those types of media. I do this on macOS, Linux, and Windows.
Not exactly, but as I poke around those folders on my desktop PC, all of them hold various things. If I didn't have a NAS they'd hold a lot more stuff too.
Browsers default into putting files in ~/Downloads, so that one gets used. DEs default into showing anything under ~/Desktop on your screen, so people disable the desktop file browser.
~/Documents is there because Freedesktop.org somehow thought they were dealing with Windows and tried to port the solution of problems Linux does not have. The other directories aren't used on any OS and only serve to annoy people.
I don’t know anyone who actually uses the Documents folder under Windows either.
I don't know anyone, other than a few tech-ignorant people who put everything on the desktop, who doesn't use them. Where else would you put stuff like music, pictures, documents, etc? These different types of files usually call for different hierarchies, so you've got to do something.
What do you do instead?
>These different types of files usually call for different hierarchies
I organize my files by subject, not file type.
My files about space stuff go in a directory called 'space'. Files about the space shuttle go in space/shuttle. This directory contains any sort of file related to the space shuttle, be it picture, video, pdf, epub, or otherwise. I don't see any utility in breaking this directory apart into ~/Pictures/shuttle, ~/Videos/shuttle, ~/Documents/shuttle... Do you really do something like that? How far do you take this "different files, different hierarchies" principle? Do pdfs and docx go in two different hierarchies too? Are pngs and jpegs separated? What's your criteria for deciding when different sorts of files get different hierarchies?
What I do is probably more similar to you than you realize. I'm not categorically placing all images in ~/Pictures, rather I'm using that directory primarily for pictures that I've taken. Each photo taking session gets its own sub-directory with the ISO 8601 date followed by a description of the event. This is extremely convenient because software I use (like raw image developers) automatically knows to look in ~/Pictures and I can access my files right away.
Likewise, I'm not using ~/Documents just for stuff with an .odt or .pdf extension. Rather, any sort of human-readable text based file goes there. I have some markdown files that have embedded images, and of course the .png or .svg files exist alongside them in the relevant directories.
~/Music is probably the best example of why the default XDG approach makes sense. You can place everything in an Artist/Album hierarchy, and most music software is written to automatically pick up and scan these directories to keep their internal libraries in sync. There's no fuss.
I don't limit myself to just the XDG directories when there's not a close fit; i also have a ~/Projects for various things I've made and ~/Programs for statically built programs that can't be installed, and so on.
Basically, I've found that the XDG approach does a pretty good job of splitting up files by purpose, allowing you to compartmentalize by avoiding having too many directories at the top level of the hierarchy.
As an aside, the desktop is a great place as a TODO-list / sticky note. Your stuff sitting there is so present and ugly, you will feel anxiety to get it sorted to a proper place.
I am more likely to do something with those files than if they are hidden away in Downloads (which is like a tempdir), or on windows, some User/Documents subfolder somewhere.
You don't know a lot of people, then - Most people, the vast majority, are tech ignorant that way.
The concept that files are cross-compatible, and usable outside the app that created them, is increasingly foreign to an online app oriented world.
My point is that that sort of person is not organizing their files at all. They're not refusing the Linux/Windows/Mac hierarchy and choosing a different one consciously, they just don't understand that the hierarchy even exists.
A lot of people running more "minimalistic" window managers like i3 or dwm or whatnot don't really have a "desktop" like Windows, macOS, GNOME, KDE, etc. have, and everything tends to just go in ~, or some subdirectory of that.
Downloads is fine, IMO. It is basically a temporary files directory. I'd be fine with changing the name to tmp, but I don't really see a reason to change the default.
Downloads is not a temporary files directory, that's what /tmp is for, I for one am using $HOME/Tmp instead of /tmp because /tmp is for whoever doesn't respect $TMPDIR big problem if you don't have much ram and you want to avoid writing to an sdcard. (assuming /tmp is tmpfs)
Okay, it's persistent storage for potentially arbitrarily large files that probably came from a browser, that you haven't gone out of your way to find another location for. You can go pretty far outside the viable scope of /tmp, but still have the files' presence there be logically temporary.
> (assuming /tmp is tmpfs)
Not a safe assumption, annoyingly. Debian doesn't use tmpfs for /tmp; I guess they expect you to use /dev/shm if you want a tmpfs.
Which is non-portable.
(Well, it exists in freebsd, but not openbsd, no clue if it exists in netbsd/minix)
I find Desktop fine, it serves a distinct purpose; Downloads doesn't seem like the greatest idea, but works well enough considering browsers. Documents/Music/Video/&c., though, I definitely concur doesn't provide any obvious value being more than one directory.
Imagine being the poor soul who had to implement Windows's "3D Objects" directory.
I’m tempted to rename "Pictures" to "2D Objects" and "Music" to "1D Objects".
> Experienced users ignore them or delete them
I wish I could! But ~/Desktop and ~/Downloads keep getting re-created by ... something :-/ I tried setting XDG_{DESKTOP,DOWNLOAD}_DIR but it seems to get ignored. Been annoying me for years.
Modify ~/$XDG_DIRNAME_DIR/user-dirs.dirs (usually ~/.config/user-dirs.dirs). You can just point them all to $HOME if you don't want any sub-directories created.
I'm fairly certain I tried this in the (distant) past but that it didn't take, but I've tried again; we'll see if it gets re-created.
>I wish I could! But ~/Desktop and ~/Downloads keep getting re-created by ... something :-/ I tried setting XDG_{DESKTOP,DOWNLOAD}_DIR but it seems to get ignored. Been annoying me for years.
Perhaps something like:
That would solve the problem, no?It wouldn’t solve the problem of those directory entries being present.
I actually use these extensively on my mac, believe it or not.
That is annoying but easily fixed with xdg-user-dirs-update eg:
My annoyance is that they create any 'helpful' directories. It's just as bad on Windows. "Download", "Pictures", "3D crap", etc....
Nope, fuck that.
This is my $home, I'll make it look how I want.
You may indeed prefer an empty canvas you have full control over.
However, creating these folders by default is actually beneficial to most users. Most users (especially the vast majority that transitioned from Windows) expect these folders to exist.
I believe the choice to not create these existed and still exists. Yet either all the distros that made that choice are history or they all made the opposite choice.
I can say the same about other choices. I vastly prefer having a TRRS headphone jack and expandable storage on a phone. These do not appear to be priorities for most people.
The choice not to create these directories basically doesn't exist, because all sorts of software will automatically create them if they don't exist, their creation isn't centrally orchestrated. A while back I was using a minimal i3-based desktop and when I launched transmission-gtk it thought I might want a "Downloads" folder, and would create it whenever it launched, even though it had been configures to download files elsewhere.
That not really true. Most desktop environments will create the directories on session startup but depending on the DE that behavior can be disabled (in xfce for example has a service in the session you can disable).
It is true that applications will create these directories if they don't exist. But you can fix this by setting the XDG user directories[0] to something that isn't in your home directory. I set them all to a subdirectory under `~/.xdg-user-dir/`, but you could just as easily set them to `/tmp`. Nothing has recreated ~/Documents on my system in years.
[0]: https://wiki.archlinux.org/title/XDG_user_directories
I was not aware of that behaviour. I agree that should not happen. Or the user should be asked if they want that. This should very much be under the control of the system and the apps the user chooses to manage the system. In a capability-based OS, apps would not be able to misbehave like this, unless granted the capability.
> This is my $home, I'll make it look how I want.
I have long since accepted that it’s not my $home after all, but rather a general purpose application config dumping ground. My actual “home” is now a top level directory used only by me.
I had no idea. Excellent tip!
I agree, but the OpenBSD did some nice things in regards to ~/Downloads
Using unveil() and pledge(), OpenBSD hides many $HOME and system directories from Firefox and chrome. Thus an errant javascript will only see things in ~/Downloads
So one nice use for these.
I like to lowercase mine so bin and projects don't look weirdly out of place. Thankfully it's totally customisable.
https://github.com/peteryates/dotfiles/blob/master/xdg/.conf...
What Rob is griping about is more analogous to $HOME/.config now, or /var/run for non-user specific stuff.
It's even worse with dotfiles nowadays than before. We have the cluttered "xdg" dotfile directories, and then a bunch of stuff makes random dotfiles/dotdirs in the home directory, and then .profiles/shellrcs/xinits that are basically required. Every single program seems to think you want a scrollback history, a configuration directory, or some random garbage to remind of you of that one time used it.
Complying with the XDG standard seems about as good as a stringly typed OS like Unix can get.
Part of which is making sure that the defaults the program chooses may be overridden by consistent environment variables.
Programs need some way to persist their own state and configuration, after all.
The fact that XDG is more honored in the breach is tiresome, but this can at least be corrected, and it's well enough put together to be a steady-state approach to wrangling dotfiles.
Windows fixed this by just having %APPDATA%, and to me, its a great idea. No .config, no .local, no .share, no .randomfile, each installed program has its directory in hidden directory, and nobody has to know what each directory is for, and sure, this is what ~/.config is for some, but I also see there's some stuff in ~/.local/share and so on.
To expand on this, when a program gets installed, it should write its binary location in a file, then if you ever erase it, a doesthisexistanymore $Config/dir/*/symlink | remove_entire_configuration_directory should be run via cron daily.
No manual cleanup, ever.
Slight correction here, XDG keys are analogous to Windows standards: https://github.com/adrg/xdg/blob/master/README.md
A long time ago I liked listening to original soundtracks for games. Winamp had a plugin that would play .psf (et al) files. There were files that contained the extracted code from games for playing the music. The plugin had a tiny builtin emulator that could run the relevant parts. No graphics or input I suppose.
One soundtrack I really liked was .hack, including the leading dot. Given that "hack" part and the overall theme of the games I assume the initial dot was quite intended. But what the creators could not have foreseen was that their soundtrack was missing from the majority of psf mirrors (but some had it obviously). I guess some part of some stack ignored dot-prefixed files when doing the mirroring.
Anyway, although I'm usually opposed to inband signalling, I am a fan of dot-prefix for hiding. Filenames already have so many special corner cases that you would have to treat them very carefully even without this one. The fact that it is embedded in the name means that any system can roundtrip hiddenness - something that cannot be said for metadata.
All my important dotfiles are symlinks to ~/local/etc, which is versioned.
I got me a copy of The Unix Programming Environment (1984) by Pike and Kernighan last year. It has plenty of hidden small design gems or anecdotes like this one. For example, why files don't need a \EOF char, it's implicit when you don't read more bytes the you have reached the end of a file...
Unix design was about making things work (1) and simple (2).
This book is online for free (three cheers for exotic spelling at archive.org).
http://files.catwell.info/misc/mirror/the-unix-programming-e...
https://archive.org/details/UnixProgrammingEnviornment
I wonder if \EOF was around due to punch cards? You maybe didn't want an implicit EOF in case you lost the last card.
EOF was around because many early file systems only recorded the number of allocated blocks per file, so some other mechanism was needed to record the precise length in bytes in cases where padding with spaces was not acceptable.
Exactly - think about a tape with multiple files on it - you want some way of knowing you reached the end of a file without having to seek all the way back to some directory.
That's an interesting history lesson! I always assumed hidden dotfiles was a conscious design decision.
It's not a history lesson. It's just baseless conjecture.
100%
I’m on the verge of abandoning the home directory. I’ll put My files elsewhere and leave home to the apps.
I thought XDG would surely solve this problem but it’s painfully clear now that most developers just don’t care. `node_modules` and `snap` are good cases in point. They didn’t even bother hiding them! LOL
I feel consternation about symlinks on Windows. I find them extremely useful, but you never know which program is going to ignore them and which program is going to follow them. They're inconsistent. Plus you can't tell by looking at files if they are symlinks or not because they have the same icon as shortcuts.
I recently managed to crash Visual Studio 2022 because I had a symlink to a drive with a hidden $RECYCLE_BIN on it.
If you copy files in Windows the system will copy the contents of symlinks, however many backup programs will not follow symlinks.
>I'm pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.
So it's just an assumption and so the title is misleading
Adding . and .. was a mistake by itself. I never needed them, but always had to write conditions to skip them when traversing the file system.
Also, dotfiles don't work well with masks because `*` doesn't match dot-files, and `.*` match all dotfiles including . and .. which causes many programs to fall into endless recursion (for example, `du -sh .*`).
Everything related to dotfiles in Linux is done wrong.
Wildcards are entirely shell-dependent so blame the shell you're using, not Linux :)
glob and fnmatch are C functions defined in POSIX though.
One way to make your config files more visible:
If you want to move and link files that you want to make visible: If you want, you can set a custom location in your system file `/etc/profile` or your user file `~/.profile`, or `~/.bashrc` or `~/.zshenv` etc. Helpful links for XDG:https://specifications.freedesktop.org/basedir-spec/basedir-...
https://wiki.archlinux.org/title/XDG_user_directories
Why move the folder? Why not just symlink config to .config? There's almost certainly some poorly written tools out there that will blow up on the symlink. Why take that risk?
You're correct, moving the folder can expose a poorly written tool, meaning a tool that isn't following the symlink and isn't using XDG_CONFIG_HOME.
To me, it's an advantage to surface the latent problems, so I can message the authors to ask them about updating. So far, all authors are good with updating.
It turns out the bulk of the issues are in internal scripts that hardcoded ~/.config instead of using XDG_CONFIG_HOME. This is an easy fix, and at the same time we shellcheck those scripts.
From a purely esthetic point of view, if you're using a graphical file manager, the symbolic link might be denoted by an ugly icon. Surely there's a solution to that, but that's the only reason I could find.
I have a directory under ~/config, ~/config/dotfiles, that just has files that are intended to be dot-prefixed and linked in the home directory. That avoids any accidental over-linking; I can just run a script that links ~/config/dotfiles/$X to ~/.$X and I'm done.
I agree that it’s annoying that so much junk accumulates in the root of your home dir.
But any consumer-facing OS will need some sort of mechanism of hiding/protecting configuration files/logs/etc. from deletion by clueless users, like how macOS has started hiding ~/Library by default.
The article points out that a better way to do this would be to designate one particular folder for configs, and then that alone could be hidden if desired. But we have what we have instead because a "convenient" hack was misappropriated.
Yes, but it also claims that hidden files in general were a mistake and mentions how Plan9 not having hidden files was learning from this mistake.
Config files are only one subset of possible dotfiles (a common one to be fair).
To some degree this is solved when using stow and a dedicated ~/dotfiles directory. No need to dig around in ~/.config
But I agree with Pike. ls -a in the home sir, becomes an unorganized mess with all those apps that don't use the xdg default .config folder.
I find the best way to read a Pike statement is to take it as a challenge to your own strongly held belief. I like .dotfiles behaviour and ls -a does not distress me. But, he makes a case it was not deliberate and probably less beneficial than we think.
So I look at that as a reframing: do I have a good rebuttal? No. I have my own muscle memory and comfort factor, but that's all. How would I think if dotfile hiding didn't happen?
Hiding . & .. is arguably as silly. It's not like they can ever not exist but why hide them?
Maybe a dumb question but why do so many apps rely on dotfiles when /etc exists? I'm not making a judgement, this is an earnest question as even though it's meant for system config files, is there any reason that all app configurations shouldn't live there?
If not in /etc, then at least /usr/etc? I'm sure there is some historical reason but some brief searching and I've not really come up with why dotfiles everywhere are preferable to a centralized configuration directory.
Apps can rely on both /etc (for system defaults) and on ~/.something (for per user defaults).
If we used only /etc, then we would need per user entries there, which would be far more difficult, esp. with network-mounted home folders, especially when those are automounted.
It would be simple with union mounts.
It would be simple now. It took quite a while to get them right. My first thought was namespaces, but they are even more recent.
The ~/.* convention worked a long time ago, far before either union mounts or namespaces were reliable.
Not that recent considering that's how plan9 works and plan9 is from the 90's. 1992[0] to be exact (apparently).
[0]: https://plan9.io/sys/doc/names.html
Even given Plan9, which I was aware of (hey, rc was my shell for a while in the early 90s), how many tries did it take to get right in Linux, even with that knowledge and experience? At least two, arguably three, and, arguably, unless namespaces, we didn’t really get there.
(On a somewhat related note, I was very excited that when HP bought Apollo they might integrate the // resource naming into HP-UX. That would have been nice. I say related, because there is a great deal of cool from other systems that died with those systems.)
Multi-user systems, where things outside $HOME are not editable, and where different users may have different configurations.
Dotfiles are a per user application config. Other users of the same system are unlikely to want my idiosyncratic zsh setup. Hence the suggestion for $HOME/cfg in the article.
I assume that because on true multi-user systems, syncing permissions between $HOME and /usr/etc/$USER may be a complete disaster.
Back ups would require you to know which paths are actually yours and where, chrooting would be problematic.
The second comment there has its merit in UNIX. However, it is moot with namespace in Plan 9. You can have different versions of the same directory structure, per process.
It's not like it's easy to manage namespaces and what all is in what namespace.
That's why some files like gitignore and clang-format start with a dot? Some old way of hiding the file?
How bizarre. I had no idea it was actually related to the . and .. files.
> Some old way of hiding the file?
An old way, but also it's still the way to hide a file, on *nix systems.
(Hence the "mistake", and the countless hours mentioned in TFA are still adding up, as people forget them when they shouldn't, or don't exclude them when they should…)
Practically the only remotely common desktop or server OS where it's not still a normal way to hide files, is Windows.
that's because they are all based on unix. or is there any remotely common OS besides windows that is not based on unix?
a few decades ago macOS, amiga and atari were rather common, which all don't follow that convention. i don't know about BeOS/haiku. but if that hides dot-files then it's because they ported a lot of unix tools for commandline use.
i could not find anything about OS/2 but i suspect that it worked like windows.
Windows is very common.
I didn't mean to imply that Windows is uncommon, but that you have to get pretty damn obscure to find another modern desktop or server OS that doesn't do the dot-equals-hidden thing. Windows is the most common on the desktop, obviously, but if you step outside it you're almost certainly in dot-is-hidden territory.
Windows is still so overwhelmingly common that it's very possible to never have occasion to touch "another" desktop or server OS. Especially if you don't go out of your way to manage servers, and doubly so if you don't live in one of the rich economies.
GP is saying "out of the many commonly used OSes, Windows is the only one which doesn't hide dot files", not that "Windows is an uncommon OS".
But it all makes sense, Windows is CP/M based, not Unix based.
Unfortunately
Windows NT was designed by former DEC employees. DEC had a rather low opinion of Unix and dotfiles definitely would have been one of the lazy hacks they disliked.
The question is why anyone feels files like .gitignore should be hidden by default from ls output - there even seems to be disagreement between versions of git whether git itself ignores such files by default (using git add e.g.), though I don't believe it does, can't check currently.
Was finally able to check on one of my repos and git 2.34.1 most definitely does ignore . files by default (both git status and git add), unless you specify --ignore for git status or --force for git add. But my .gitignore (either for the clone or global core.excludesfile) doesn't specify that . files should be ignored, so why is it doing so?
I can't see how to do a git add to add all new files, including dot files, except those in .gitignore.
Dotfile synchronization is a necessary evil. At work, we have a dotfile synchronization system that contains an infrastructure-level yml file of what to save, what are history files, and what to exclude. Every time a modification is made, it saves a revision. You can have multiple dotfile environments also that completely swaps them out. On the work laptop, I have google drive saving a .dotfiles dir with symlinks into them.
>I was wondering why this post had so many comments. Then I found out it was posted on proggit/HN. The discussions there were hardly interesting, however.
For the curious (was a bit annoying to find since gplus is down now and all links redirect to a different subdomain):
https://news.ycombinator.com/item?id=4331855
Haha, the first comment is of me throwing shade on HN, oh geez. Now I'm curious to read the old HN thread that I thought was so disappointing: https://news.ycombinator.com/item?id=4331855
Dot files are much better than hidden files on Windows (and before DOS). There you need additional attributes to hide files.
And hidden files seem to be an important thing. Some things you need to hide from the end user.
>Some things you need to hide from the end user. De-emphasize - maybe, hide - no. Garbage pile that is average Linux home directory is pretty much result of 'out of sight - out of mind' practice, enabled by dotfiles being hidden. I wish Linux settled on single `~/System/` directory, mirroring `/` structure.
On the other hand, dotfiles convention is useful for meta/"parallel" data, `.git` being prime example - something tighly linked with abitrary file structure.
Sure, the dot files in ~/ are a mess.
It would be so much cleaner if the convention would've bin similar to /var, /etc, and so on. ~/.var/cache or ~/.etc/
this post is a good example of how un*x hippies with their fried brains from too much drugs* realize one unit of common sense after aeons of contemplation. just like with generics in go. in fact, go itself is just decades of C users very slowly conceding to the idea of basic hygiene. it took them decades to come up with something which is essentially just what was already available in the 90s (pascal, fortran, ada, basic, later java 1.2 or so which also had green threads), but with stuff done a certain way so the boomers can be like "oh see, we did this minor syntactic change THIS way, checkmate, pascal"), in other words they want to have their say in everything
* im not joking, this is what un*x culture appears to essentially boil down to. boomers were all hippies in the 70s despite presenting themselves as "austere" and "mature" "wise" individuals now, which had the unfortunate side effect of too much drugs. further research is needed to clarify whether brain damage is caused by drugs or un*x
any encoding in file names at all is a mistake. this includes character encodings too, which may sound like a non-sequitur but its not. in fact, files should only have unique identities, not names. names should be metadata. folder hierarchy should just be a user defined data structure that can reference these said "files". anyone who tries developing their own OS without copying extremely idiosyncratic un*x ideas and without being bogged down by the overhead of assembly language or C quickly learns this. an example of youngins discovering this is IPFS (and subsequently implementing it on top of broken un*x)
This is an AI generated comment, right?
You didn't get a job at freaking AT&T Bell Labs by being a pot smoking hippie. They were the squarest of square professionals--suit, tie, the whole bit.
no, it just doesn't fit into your two categories of posters here on HN
lmao based
just put them all in ~/configs ... many apps still pollute ~/
A happy little accident
However Apple does it is best.
alias ls='ls -a'
I tend to always start with:
How interesting (: I find myself typing a lot of `ls -dl /a/b/d/*` these days.
What a strange fingerprint I imagine all our collective accumulated behaviors project about our past experiences.
It's not just ls though, the shell hides them too right? echo * won't show them.
Well, I was being a little flippant dismissing it as largely a complaint about things not showing up in the terminal, but it's obviously deeper than that, yes.
I think this overall situation is most acutely a lack of standard. Someone up in the nosebleed section rightly pointed out that there should be some environment variable for a config to be written to, and I agree. I also want to stress that there should be the inclusion of a single variable on top of that, that all programs should use when constructing the path to their configuration. USER_CONFIG_DIR or something equally arguable should, if unset, default to $HOME, and if set, be used as the root to construct whatever structure therein is desired, also able to be changed with an environment variable specific to that program. Be it filename or further subdirectory, first using a single reference that is /not overloaded/, like HOME, but defaults to HOME if unset.
Very good point and arguably a bigger problem. Worth putting `shopt -s dotglob` in .bashrc to avoid. How about other shells?
First of all, “A lot of other lazy programmers”. That’s cringe. I don’t care who you are, that’s not a way to talk.
Secondly, I think we have all been there. We introduce a bug in an interface/API but then the users of it start depending on this behaviour and before you know it it’s a feature and it’s impossible to fix without breaking the world.
He's obviously a very smart guy and a pioneer and a much more accomplished person than me (or most of us), but Rob Pike seems to make a habit of arrogantly talking down about other people's intelligence or skill:
Re: Go “The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.” -- Rob Pike
I agree it's really a cringe way of approaching people. In this (dotfile) case there may have been some self-deprecation in the comment though, so it's frankly hard to get too miffed about it.
I think he very accurately described the problem Go is trying to solve which is how do you make new grads productive?
In my experience at AWS people that graduated didn't understand the languages we used. This leads to a bunch of problems, and makes it's easy to write very bad, unmaintainable code. Lots of needless class hierarchies in Java, or a misunderstanding of `this` in JavaScript. Go solves the problem perfectly.
Never had that experience at Google. New grads and interns always seemed sharp. Might just be the superior quality of education out of U of Waterloo, or the hiring bar at our site, or both. I enjoyed working with them.
With new grads, I generally didn't have to teach coding knowledge. It was more about transmitting wisdom around best practices, estimation, architecture, etc.
If anything, to me Go makes a lot of things harder that could be simpler. Lack of generics means boilerplate. Eschewing test frameworks means boilerplate. The structural typing stuff is exotic (but neat I guess) and not familiar to most new programmers. The actor / CSP type concurrency stuff is also exotic and not familiar to most new programmers. I don't see how it's a language for new programmers.
It also seems very much like a rehash of the concepts they had in Limbo so I actually don't find it honest for him to be claiming it was designed for this purpose (to simplify programming). Instead it feels very much like they had a hammer in search of a nail, an itch they've been scratching since the Plan9 days. And that's fine. I just don't buy this pitch about its market niche and the stuff about the relative intelligence of programmers is gratuitous. I've seen some very smart new grads produce amazing Rust code for example.
Go has generics now, and they're great.
Anyway, the idea isn't that everyone already knows Go because it uses familiar features for concurrency, typing, etc.
The idea is that Go is such a simple language, that anyone with solid CS experience can pick it up in a short amount of time and start being productive with it.
Calling someone out (Rob Pike or not) for "A lot of other lazy programmers" is cringe. It's clear and accurate communication, and shouldn't need to be workshopped to avoid offending people determined to be offended.
Good point. Rob Pike has nothing interesting to add here. Just an old white dude making us cringe. It's like come on grandpa, its the 2020s, we've all been there.