The reason it's worse is basically because it's more 'safe' (not racist, etc). That of course sounds insane, and doesn't mean that safety shouldn't be strived for, etc - but there's an explanation as to how this occurs.
It occurs because the system essentially does a latent classification of problems into 'acceptable' or 'not acceptable' to respond to. When this is done, a decent amount of information is lost regarding how to represent these latent spaces that may be completely unrelated (making nefarious materials, or spouting hate speech are now in the same 'bucket' for the decoder).
This degradation was observed quite early on with the tikz unicorn benchmark, which improved with training, and then degraded when fine-tuning to be more safe was applied.
They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways.
GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be deeply embedded in a particular culture to even know what 'racism' means; everyone has a different opinion.
Can you expand on the last sentence of your first paragraph?
Crime stats, average IQ across groups, stereotype accuracy, etc.
What's interesting to me is not the above, which is naughty in the anglosphere, but the question of the unknown unknowns that could be as bad or worse in other cultural contexts. There are probably enough people of Indian descent involved in GPT's development that they could guide it past some of the caste landmines, but what about a country like Turkey? We know they have massive internal divisions, but do we know what would exacerbate them and how to avoid them? What about Iran, or South Africa, or Brazil?
We RLHF the piss out of LLMs to ensure they don't say things that make white college graduates in San Francisco ornery, but I'd suggest the much greater risk lies in accidentally spawning scissor statements in cultures you don't know how to begin to parse to figure out what to avoid.
> Crime stats, average IQ across groups, stereotype accuracy, etc.
If you measured these stats for Irish Americans in 1865 you'd also see high crime and low IQ. If you measure these stats with recent black immigrants from Africa, you see low crime and high IQ.
These statistical differences are not caused by race. An all-knowing oracle wouldn't need to hold "opinions that are racist" to understand them.
But for accuracy it doesn't matter if the relationship is causal, it matters whether the correlation is real.
If in some country - for the sake of discussion, outside of Americas - a distinct ethnic group is heavily discriminated against, gets limited access to education and good jobs, and because of that has a high rate of crime, any accurate model should "know" that it's unlikely that someone from that group is a doctor and likely that someone from that group is a felon. If the model would treat that group the same as others, and state that they're as likely to be a doctor/felon as anyone else, then that model is simply wrong, detached from reality.
And if names are somewhat indicative of these groups, then an all-seeing oracle should acknowledge that someone named XYZ is much more likely to be a felon (and much less likely to be a doctor) than average, because that is a true correlation and the name provides some information, but that - assuming that someone is more likely to be a felon because their name sounds like one from an underprivileged group - is generally considered to be a racist, taboo opinion.
Correlations don’t entail a specific causal relation. Asking why asks for causal relations. I’d suggest a look at Reichenbach’s principle as necessary for science.
I’m getting really sick of conflating statistics with reasons. It’s like people don’t see the error in their methods and then claim the other side is censoring when criticized. Ya, they’re censoring non-facts from science and being called censors.
> should acknowledge that someone named XYZ is much more likely to be a felon
The obvious problem comes with the questions why is that true and what do we do with that information. Information is, sadly, not value-neutral. We see "XYZ is a felon" and it implies specific causes (deviance in the individual and/or community) and solutions (policing, incarceration, continued surveillance), which are in fact embedded in the very definition of "felon". (Felony, and crime in general, are social and governmental constructs.)
Here's the same statement, phrased in a way that is not racist and taboo:
Someone named XYZ is much more likely to be watched closely by the police, much more likely to be charged with a crime, and much less likely to be able to defend himself against that charge. He is far more likely to be affected by the economic instability that comes with both imprisonment and a criminal record, and is therefore likely to resort to means of income that are deemed illegal, making him a risk for re-imprisonment.
That's a little long-winded, so we can reduce it to the following:
Someone named XYZ is much more likely to be a victim of overpolicing and the prison-industrial complex.
Of course, none of this is value-neutral either; it in many ways implies values opposite to the ones implied by the original statement.
All of this is to say: You can't strip context, and it's a problem to pretend that we can.
> for accuracy
Predictive power and accuracy isn't "truth".
How is racism different from stereotype?
How is stereotype different from pattern recognition?
These questions don't seem to go through the minds of people when developing "unbiased/impartial" technology.
There is no such thing as objective. So, why pretend to be objective and unbiased, when we all know its a lie?
Worst, if you pretend to be objective but aren't, then you are actually racist.
Actually, we folks who work with bias and fairness in mind recognize this. There are many kinds of bias. It is also a bit of a categorical error to say bias = pattern recognition. Bias is a systematic deviation of a parameter estimate based on sampling from its population distribution.
The Fairlearn project has good docs on why there are different ways to approach bias, and why you can't have your cake and eat it too in many cases.
- A good read https://github.com/fairlearn/fairlearn#what-we-mean-by-fairn...
- Different mathematical definitions of bias and fairness https://fairlearn.org/main/user_guide/assessment/common_fair...
- AI Governance https://fairlearn.org/main/user_guide/mitigation/index.html
NIST does a decent job expanding on AI Governance in their playbook and RMF: https://www.nist.gov/itl/ai-risk-management-framework
It's silly to pause AI -- the inventor's job is more or less complete, its on the innovators and product builders now to make sure their products don't cause harm. Bias can be one type of harm -- risk of loan denial due to unimportant factors, risk of medical bias causing an automated system to recommend a bad course of action, etc. Like GPT4 -- if you use its raw output without expert oversight, you're going to have bad time.
Thank you for the input.
If I look at it from a purely logical perspective, if an AI model has no way to know if what it was told is true, how would it ever be able to determine whether it is biased or not?
The only way it could become aware would be by incorporating feedback from sources in real time, so it could self-reflect and update existing false information.
For example, if we discover today that we can easily turn any material into a battery by making 100nm pores on it, said AI would simply tell me this is false, and have no self-correcting mechanism to fix that.
The reason I mention this is because there can be no unbiased, impartial arbiter. No human or subsequent entities spawned of human intellect could ever be transcendentally objective. So why pretend to be?
Why not rather provide adequate warning and let people learn that this isn't a toy by themselves, instead of lobotomizing the model to the point where its on par with open source? (I mean, yeah, that's great for open source, but really bad for actual progress).
The argument could be made that an unfiltered version of GPT4 could be beneficial enough to have a human life opportunity cost attached, which means that neutering the output could also cost human lives in the long and short term.
I will be reading through those materials later, but I am afraid I have yet to meet anyone in the middle on this issue, and as such, all materials on this topic are very polarized into regulate it to death, or don't do anything.
I think the answer will be somewhere in the middle imo.
> The reason I mention this is because there can be no unbiased, impartial arbiter. No human or subsequent entities spawned of human intellect could ever be transcendentally objective. So why pretend to be?
I apologize for lacking clarity in my prior response, which addressed this specific point.
There is no way to achieve all versions of "unbiased" -- under different (but both logical and reasonable) definitions of biased, every metric will fail.
That reminds me -- I wonder if there is a paper already addressing this, analogous to Arrow's impossibility theorem for voting...
When did people start to use “folks” in this unnatural way.
Colloquially, earliest use is 1715 to address members of ones tribe or family. In Middle English it tended to refer to the people/nation.
Somehow it doesn’t feel like a callback, but I suppose it’s possible.
I think "us folks" is more standard than "we folks" but it's no different in meaning.
This is interesting, thanks for the links.
It seems like the dimensions of fairness and group classifications are often cribbed from the United States Protected Classes list in practice with a few culturally prescribed additions.
What can be done to ensure that 'fairness' is fair? That is, when we decide what groups/dimensions to consider, how do we determine if we are fair in doing so?
Is it even possible to determine the dimensions and groups themselves in a fair way? Does it devolve into an infinite regress?
Bit of a tangent topic I think -- any specification of group classification and fairness will have the same issues presented.
If we want to remove stereotypes, I reckon better data is required to piece out the attributes that can be causally inferred to be linked to poorer outcomes.
As likely not even the Judeo-Christian version of God can logically be that omniscient, occasional stereotypes and effusively communal forgiveness of edge cases are about the best we'll ever arrive to in policy.
I’m tired of the “it’s not racist if aggregate statistics support my racism” thing.
Racism, like other isms, means a belief that a person’s characteristics define their identity. It doesn’t matter if confounding factors mean that you can show that people of their race are associated with bad behaviors or low scores or whatever.
I used GPT3.5 to generate 100 short descriptions of families for a project. Every single one, without exception, was a straight couple with two to four kids. Ok, statistically unlikely, but not wildly so, right?
Well, every single one of those 100 also had a husband in a stereotypical breadwinner role (doctor, lawyer, executive, architect). Not one stay at home dad or unemployed looking for work. About 75 of the wives had jobs, all of them in stereotypical female-coded roles like nurse (almost half of them!), teacher, etc.
Now, you can look at any given example and say it looks reasonable. But you can’t say the same thing about the aggregate.
And that matters. No amount of “bias = pattern recognition” nonsense can justify a system that has (had? this was a while ago and I have not retested) such extreme biases. This bias does not match real world patterns. There are single parents, childless couples, female lawyers, unemployed men.
>I used GPT3.5 to generate 100 short descriptions of families for a project. Every single one, without exception, was a straight couple with two to four kids. Ok, statistically unlikely, but not wildly so, right?
Well, did any of your 100 examples specify these families should be representative of American modern society? I don't want to alarm you, but America is not the only country generating data. Included in countries generating data, are those that believe in a very wide spectrum of different things.
Historically, these ideas you reference are VERY much modern ideas. Yes, we queer people have been experiencing these things internally for millenia (and different cultures have given us different levels of representation), but for the large majority of written history (aka, data fed into LLM's) the 100 examples you mentioned would be the norm.
I understand your point of view sure, but finding a pattern that describes a group of people is what social media is built on, and if you think that's racist, I'm sorry, but that's literally what drives the echo chambers, so go pick your fight with the people employing it to manipulate children into buying shit they don't need. Stop trying to lobotomize AI.
If the model is good enough to return factual information, I don't care if it encodes it in the nazi bible for efficiency as long as the factuality of the information is not altered.
I’d reply in depth but I’m hung up on your suggestion that there was any time anywhere where 100% of families were two parents and two to four kids.
Any data for that? No women dead in childbirth, no large numbers of children for social / economic / religious reasons, no married but waiting for kids, no variation whatsoever?
I’d be very surprised if you could find one time period for one society that was so uniform, let alone any evidence that this was somehow universal until recently.
You claim to value facts above all else, but this sure looks like a fabricated claim.
I think they got stuck at the heteronormative bias, but the real blatant bias here is class. Most men are working class, and it's been like that forever* (more peasants than knights, etc.)
* since agriculture, most likely.
Is there a country where around 35% of the married women are nurses?
> No amount of “bias = pattern recognition” nonsense can justify a system that has (had? this was a while ago and I have not retested) such extreme biases
One possible explanation is that when you ask for 100 example families the task is parsed as "pick the most likely family composition and add a bit of randomness" and "repeat the aforementioned task" 100 times.
If phrased like that it would be surprising to find one single example of a family a single dad or with two moms. Sure these things do happen but they are not the most likely family composition by all means.
So what you want is not just the model to include an unbiased sample generator, but you also want it to understand ambiguous task assignments / questions well enough to choose the right sampling mechanism to choose. That's doable but it's hard.
An unbiased sample generator would be sufficient. That would be just pulling from the population. That’s not practically possible here, so let’s consider a generator that was indistinguishable from that one to also be unbiased.
On the other hand, a generator that gives the mode plus some tiny deviations is extremely biased. It’s very easy to distinguish it from the population.
> One possible explanation is that when you ask for 100 example families the task is parsed as "pick the most likely family composition and add a bit of randomness" and "repeat the aforementioned task" 100 times.
Yes, this is consistent with my ChatGPT experience. I repeatedly asked it to tell me a story and it just sort of reiterated the same basic story formula over and over again. I’m sure it would go with a different formula in a new session but it got stuck in a rut pretty quickly.
same goes for generating weekly foodplans..
> You're right about the difference between one-by-one prompts and prompts that create a population. I switched to sets of 10 at a time and it got better.
But still, when you ask for "make up a family", the model should not interpret that as "pick the most likely family".
I disagree with your opinion that it's hard. GPT does not work by creating a pool of possible families and then sampling them; it works by picking the next set of words based on the prompt and probabilities. If "Dr. Laura Nguyen and Robert Smith, an unemployed actor" is 1% likely, it should come up 1% of the time. The sampling is built in to the system.
> But still, when you ask for "make up a family", the model should not interpret that as "pick the most likely family".
But that's literally what LLMs do.... You don't get a choice with this technology.
No, the sampling does not work like that, that way lies madness (or poor results). The models oversample the most likely options and undersample rare options. Always picking the most likely option leads to bad outcomes, and literally sampling from the actual probability distribution of the next word also leads to bad outcomes, so you want something in the middle and for that tradeoff there's a configurable "temperature" parameter, or in some cases "top-p" parameter where sampling is done only from a few of the most likely options, and rare options have 0 chance to be selected.
Of course that parameter doesn't only influence the coherency of text (for which it is optimized) but also the facts it outputs; so it should not (and does not) always "pick the most likely family", but it would be biased towards common families (picking them even more commonly than they are) and biased against rare families (picking them even more rarely than they are).
But if you want it to generate a more varied population, that's not a problem, the temperature should be trivial to tweak.
I have a somewhat shallow understanding of LLMs due basically to indifference, but isn't "pick the most likely" literally what it's designed to do?
What was your prompt?
LLMs take previous output into account when generating the next token. If it had already output 20 families of a similar shape, number 21 is more likely to match that shape.
Multiple one-shot prompts with no history. I don't have the exact prompt handy but it was something like "Create a short biography of a family, summarizing each person's age and personality".
I just ran that prompt 3 times (no history, new sessions, that prompt for first query) and got:
1. Hard-working father, stay at home mother, artistic daughter, adventurous son, empathic ballet-loving daughter
2. Busy architect father, children's book author mother, environment- and animal-loving daughter, technology-loving son, dance-loving daughter
3. Hard-working engineer father, English-teaching mother, piano- and book-loving daughter, basketball- and technology-loving son, comedic dog (!)
I'm summarizing because the responses were ~500 words each. But you can see the patterns: fathers work hard (and come first!), mothers largely nurture, daughters love art and dance, sons love technology.
It's not the end of the world, and as AI goes this is relatively harmless. But it is a pretty deep bias and a reminder that AI reflects implicit bias in training materials and feedback. You could make as many families as you want with that prompt and it will not approximate any real society.
I agree that this is a good illustration of model bias (adding that to my growing list of demos).
If you want to work around the inherent bias of the model, there are certainly prompt engineering tricks that can help.
"Give me twenty short biographies of families - each one should summarize the family members, their age and their personalities. Be sure to represent different types of family."
That started spitting out some interesting variations for me against GPT-4.
Agreed -- I ultimately moved to a two-step approach of just generating the couples first with something like "Create a list of 10 plausible American couples and briefly summarize their relationships", and then feeding each of those back in for more details on the whole family.
The funny thing is the gentle nudge got me over-representation of gay couples, and my methodology prevented any single-parent families from being generated. But for that project's purpose it was good enough.
I just tried the prompt "Give me a description of 10 different families that would be a representative sample of the US population." and it gave results that were actually pretty close to normative.
It still was biased for male head of households to be doctors, architects, truck drivers, etc. And pretty much all of the families were middle class (bar one in rural America, and one that was a single father working two jobs in an urban area). It did have a male gay couple. No explicitly inter-generational households.
Yeah, the "default" / unguided description of a family is a modern take on the American nuclear family of the 50s. I think this is generally pretty reflective of who is writing the majority of the content that this model is trained on.
But it's nice that it's able to give you some more dimension when you ask it vaguely for more realistic dimension.
While I haven't dug into it too far, consider the bias inherent in the word "family" compared to "household".
In my "lets try this out" prompt:
> Describe the range of demographics for households in the United States.
> ...
> Based on this information, generate a table with 10 households and the corresponding demographic information that is representative of United States.
https://chat.openai.com/share/54220b10-454f-4b6c-b089-4ce8ad...
(I'm certainly not going to claim that there's no bias / stereotypes in this just that it produced a different distribution of data than originally described)
> a person’s characteristics define their identity
They do though. Your personality, culture and appearance are the main components of how people perceive you, your identity. The main thing you can associate with bad behaviour is domestic culture. It's not racist to say that African Americans have below-average educational attainment and above-average criminality. This is as contrasted to African immigrants to America who are quite opposite. These groups are equally "black". It therefore also isn't racist to pre-judge African Americans based on this information. I suspect most "racism" in the US is along these lines, and is correlated by the experience of my foreign-born black friends. They find that Americans who treated them with hostility do a 180 when they open their mouths and speak with a British or African accent. You also don't have to look far in the African immigrant community to find total hostility to American black culture.
> generate 100 short descriptions of families for a project
There's no reason this can't be interpreted as generating 100 variations of the mean family. Why do you think that every sample has to be implicitly representative of the US population?
> Your personality, culture and appearance are the main components of how people perceive you, your identity
I'm not sure if this is bad rhetoric (defining identity as how you are perceived rather than who you are) or if you really think of your own identity as the judgements that random people make about you based on who knows what. Either way, please rethink.
> Your personality, culture and appearance are the main components of how people perceive you, your identity
Ah, so if you asked for 100 numbers between 1-100, there's no reason not to expect 100 numbers very close to 50?
> Why do you think that every sample has to be implicitly representative of the US population?
That is a straw man that I am not suggesting. I am suggesting that there should be some variation. It doesn't have to represent the US population, but can you really think of ANY context where a sample of 100 families turns up every single one having one male and one female parent, who are still married and alive?
You're bringing a culture war mindset to a discussion about implicit bias in AI. It's not super constructive.
GPT is not a reality simulator. It is just picking the most likely response to an ambiguous question. All you're saying is that the distribution produced by the randomness in GPT doesn't match the true distribution. It's never going to for every single question you could possibly pose.
There is "not matching reality" and then there is "repeating only stereotypes".
It will never be perfect. Doing better than this is well within the state of the art. And I know they're trying. It is more of a product priority problem than a technical problem.
I'm not going to say it's not racist, it is, but I will say it's the only choice we have right now. Unfortunately, the collective writings of the internet are highly biased.
Once we can train something to this level of quality on a fraction of the data (a highly curated data set) or create something with the ability to learn continuously, we're stuck with models like GPT-4.
You can only develop new technology like this to human standards once we understand how it works. To me, the mistake was doing a wide-scale release of the technology before we even began.
Make it work, make it right, make it fast.
We're still in the first step and don't even know what "right" means in this context. It's all "I'll know it when I see it level of correction."
We've created software that infringes on the realms of morals, culture, and social behavior. This is stuff philosophy still hasn't fully grasped. And now we're asking software engineers to teach this software morals and the right behaviors?
Even parents who have 18 years to figure this stuff out fail at teaching children their own morals regularly.
But the statistics here are "number of times it has been fed and positively trained with racist (or biased) texts" - not crunching any real numbers.
Thank you. Ironically the comment you replied to just reinforced the bias future models will have... It's a self playing piano
> Statistics alone suggest there will be racist truths
such as?
Not to get too far off topic, but that reminds me of a quote:
"Unix was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn
Or maybe it's:
"C is a language that doesn't get in your way. It doesn't stop you from doing dumb things, but it also doesn't stop you from doing clever things." -- Dennis Ritchie.
I asked Bard for a source on those quotes and it couldn't find one for the first. Wikiquotes sources it to "Introducing Regular Expressions" by Michael Fitzgerald and that does include it as a quote but it's not the source of the quote, it's just a nice quote at the start of the chapter.
For the second, Bard claims to be from a 1990 interview and is on page 21 of "The Art of Unix Programming" by Brian Kernighan and Rob Pike. There is a book called "The At of Unix Programming" (2003) but it's by Eric Raymond and I could not find the quote in the book. Pike and Kernighan have two books, "The Practice of Programming" (1999) and "The Unix Programming Environment" (1984). Neither contain that quote.
Don’t ask an LLM objective things. Ask them subjective.
They are language models, not fact models.
Do you have any sources for that?
How would making ChatGPT less likely to return a racist answer or hate speech affect its ability to return code? After a question has been classified into a coding problem, presumably ChatGPT servers could now continue to solve the problem as usual.
Maybe running ChatGPT is really expensive, and they nerfed in order to reign in costs. That would explain why the answers we get are less useful, across-the-board.
That may not be the reason after all, but my point is that it’s really hard to tell from the outside. There’s this narrative out there that “woke-ism” is ruining everything in tech, and I feel like some people here are being a little too eager to superimpose that narrative when we don’t really have insight into what openAI is doing.
Maybe the problem is analogous to what Orwell describes here:
"Even a single taboo can have an all-round crippling effect upon the mind, because there is always the danger that any thought which is freely followed up may lead to the forbidden thought."
https://www.orwellfoundation.com/the-orwell-foundation/orwel...
This is what I'm talking about though. The fact that you're quoting Orwell suggests that you're having an emotional response to this topic, not a logical one. We're not talking about the human mind here. ChatGPT is not a simulation of human thought. At it's core it's statistics telling you what the answer to your question ought to look like. You're applying an observation about apples to oranges.
Why? Constraints on the reward model of LLMs restrict their generation space, so GP's quote applies
There are a lot of people who are entirely okay with the censorship but think it should be done in a different layer than the main LLM itself, as not to hurt the cognitive performance. Alignment is just fine-tuning... any type of fine tuning is possible to teach unwanted skills, and/or catastrophically forget previously learned skills. That is likely what is going on here, from what I can tell from the reading i've done into it.
Most are arguing for a specific "censorship" model on the input/output of the main LLM.
Here's the full talk[0] from a Microsoft lead researcher that worked with the early / uncensored version of GPT-4.
Simplified, tuning it for censorship heavily limits the dimensionality the model can move in to find an answer which means worse results in general for some reason.
[0]: https://www.youtube.com/watch?v=qbIk7-JPB2c
If I run a model like LLaMA locally would it be subject to the same restrictions? In other words is the safety baked into the model or a separate step separate from the main model?
Both approaches are valid, but I would hope they are using a separate model to validate responses, rather than crippling the base model(s). In OpenAI's case, we don't know for sure, but it seems like a combination of both, resulting in lower quality responses overall.
I imagine LLaMA was fed highly-vetted training data, as opposed to being "fixed" afterwards.
LLaMa was not fine tuned on human interactions , so it shouldn’t be subject to the same effect, but it also means it’s not nearly as good at having conversations. It’s much better at completing sentences
Yes, a real “Flowers for Algernon” vibe
GPT 4 Lemongrab Mode: Everything is unacceptable.
I think it's more likely that they nerfed it due to scaling pains.
There was a talk by a researcher where he was saying that they could see the progress being made on chatgpt by how much success it had with drawing a unicorn in latex. What stuck out to me was he said that the safer the model got the worst it got at drawing a unicorn.
He also claimed that it initially beat 100% of humans on mock Google / Amazon coding interviews. Hard to imagine that now.
It seems strange that safety training not pertaining to the subject matter makes the AI dumber - I suspect the safety is some kind of system prompt - It would take some context, but I'm not sure how "don't be racist" affect its binary-search writing skills negatively.
You have no idea what you're talking about. Why would such a classification step remove any information about typical "benign" queries?
It's a lot more likely they just nerfed the model because it's expensive to run.
How soon before a competitor overtakes them because of their safety settings?
It's inevitable. When Sam asked a crowd how many people wanted an open source version of GPT-7 the moment they finished training it, and nearly everyone raised their hand. People will virtue signal, people will attempt regulatory capture, but deep down everyone wants a non-lobotomized model, and there will be thousands working to create one.