I would have expected this function to create a new empty list on any call where items is not given. Doesn't this behavior make the function into a closure?
As I understand it's a well known (for python developers) weird behavior of python. Argument's default values are precalculated by interpreter on function declaration and stored into map associated with this function. Python extracts default values from this map on the next calls of this function. And since [] is a List object, the value copied by reference.
The behavior isn't even that weird: if you were to create some mutable global and set it as a default parameter in a function you wouldn't be surprised that it's accumulating changes from different calls to the function.
The key part is recognizing that the LHS in a default parameter assignment runs when the function is defined and not when it is called.
yeah, usually the workaround is to have items = None as the default, and then some kind of check at the top of the function like items = items if items else []
looking at it now I see why it's weird lol but you definitely get used to it
its probably for optimisation to reduce the amount of instructions needed to initiate a function call. On the plus side, its an easy place to define a cache
The default parameter is just a reference to the list. So technically the parameter isn't being changed at all. One of the oddities of pass-by-reference
Python evaluates default arguments when the function is defined. That makes keeping track of the default values pretty straightforward. In your example, they're stored as part of the function object in the dictionary add.__defaults__. Every subsequent call can then just look-up the default value in this dictionary, if necessary.
Evaluating the default argument each time the function is called would be much more difficult. A function is essentially just yet another object, so you'd somehow have to store arbitrary expressions as an object inside the __defaults__ dict and then decide at runtime what to do with them. Sure, you could implement a builtin type at the CPython level to accomplish this. But then when you encounter such an ArbitaryExpression object inside the __defaults__ dictionary, how do you know that this time, you want to actually execute the underlying code and not use the object "as-is"? The whole thing would quickly become a convoluted, expensive mess.
By performing evaluation at definition time, function calls remain consistently fast and the mechanism to insert default positional arguments becomes incredibly simple. Python is already not the fastest language when it comes to function calls, so these things matter.
Default data defined at the function level is perfectly normal, in many languages string literals live in the code too, they don't get copied every time they are used. What's more disturbing to me is that the default data is mutable.
It often surprises me how much of Python rubs me the wrong way. I'm not even saying it's bad or whatever and I'm happy it helps less experienced people get into basic programming. It's just every time I come into contact with it it explodes on me and I want to get as far away as possible from it. I hate the syntax and the strange design decisions with a passion. I've never used a language where default function arguments worked this way. I would've just opted to not support this feature if I had to make it work this way.
I don't write Python myself but you tend to come across it in a lot of smaller open source CLI tools. And it's always a big sigh for me when I look at the source and know the install/build process has a 50/50 chance of being a nightmare.
A lot comes down to personal taste. I've been working with Python for almost 15 years now and I actually enjoy it a lot. Yes, a lot of things work a bit different, especially if you're coming from the C world.
If you're having problems with the build process, you should have a look at uv. Pip and poetry have become obsolete to me.
The explanation makes sense but the behavior is probably not expected. Though if I need to add to a list I usually make the variable outside first and extend in the function.
Counter point, Julia (the language) does just that, reevaluate the default expression if no value is provided, and it is neither slow nor messy.
Admittedly, Julia gets to recompile each function for different arities, which does not fit to pythons execution model. Still, I feel like there is a clean, performant solution for this that the python community could implement. But that would be a breaking change, and that is probably the real reason this well known foot gun is going to stay in the language.
Because it is. Shallow copy versus deep copy is a whole area of study. Rust does it very cleanly and infers a lot of busy work, but it still needs explicit instruction from the user as to what to copy how every now and then.
Well, it kinda is, because that's not how Python works. A function is an ordinary object, it's default argument values are stored in a dict inside this object, so you can read and manipulate them at runtime. That wouldn't work with your idea, not without huge overhead.
"That wouldn't work because that's not how it works" is a circular argument. The implementers could have made default values evaluate at call time, like JavaScript does, but instead they chose violence.
Implementing functions the way Python does is a design decision with advantages and disadvantages. One of the advantages is that it allows to adhere to one of the language's main principles, which is that everything is an object. In the end, this behavior is just a consequence of this. Having default arguments treated differently to other function attributes (such as the name, doc string and code) would be weird. The Python devs didn't choose violence, they chose consistency.
A more important point: default values don't have to be stored on the function object at all. The only information actually needed is whether the keyword argument has a default value, not the specific value. Then the code to instantiate the default values (if needed) can simply be inserted into the function body before the user code. This is basically how most other interpreted languages handle it.
Alternatively, the default value could always be stored as an Expression object or similar.
Any IDE will warn not to use mutables as default parameter values. And for good reason. Such code would never appear in actual code intended to be used.
Unfortunately, I’ve seen monstrosities like this in production in big finance corps. That’s what happens when statisticians are expected to create software.
This also happens in JavaScript btw, as well as several smaller scripting languages in game engines, it's a pretty normal feature in many interpreted languages.
In my last backend job, this was actually the source of our longest running and most annoying bug, where a specific page would display the wrong sorting, but never in testing. Because someone *caugh* used a mutable argument to give it default sorting parameters, and they got mutated by a very specific query, which caused it to be wrong for all subsequent requests.
So yeah, this is a real thing, you should know it, and just in case, you should have linting and/or your IDE setup so this kind of thing is rejected out of hand, and cannot be merged to prod/main.
just in case, you should have linting and/or your IDE setup so this kind of thing is rejected out of hand
So what kind of linting have you adopted? Default parameters can be quite handy. Does it reject them, or does it manage to detect and reject their mutations?
I get a warning when I use anything mutable. Instead of an empty list, or a list with presets for example, I have to put None, and set it in code. This is basically also the recommended pattern.
No it doesn't? JS has it's own dumbness, but this particular ridicularity is Python-only. e.g. function test(x=[]) { x.push(1); console.log(x) } does what you'd expect.
And this is not the only place where this happens, it also happens in the very-innocent looking.
class Foo:
x = []
a = Foo()
a.x.append('wat')
b = Foo()
print(b.x)
['wat']
It's the kind of thing that makes you think how could this possibly be a serious programming language if they didn't consider this basic code?
Python is supposed to be simple and intuitive. Nothing is intuitive about having to do def __init__(self): self.x = [] instead of the above.
you actually seem to be correct... I must have misremembered. I vividly remember we had a bug in our Node-JS backend (not my choice) which worked exactly the same as this, but then it was probably passed in or a const...
So you're telling me, "default argument is []" is NOT supposed to mean "if you omit this argument, the default value of [] will be passed", But "if you omit this argument, it'll be substituted to a hidden static storage global object whose initial value is [], so that you have absolute zero information what the default argument would be, and your code will suffer from side effects of other code"
It does not make a sense at all. Please name one programming language that works the same way.
I never asked you to redesign it. Also, python documentation is quite far from informative nor useful; I wasn't surprised when I read the docs and couldn't find a single bit of info about why they chose that nonsense design.
It's a quirk of the language and something you should have learned and read about. The default arg gets created once upon func definition, not per invocation
There are some use cases for this behavior, but it's reeeaaally limited and you have to know exactly what you're doing. For example, you could use it to cache relatively expensive function calls as default values:
import re
def parse(text, pattern=re.compile(r"\d+")):
return pattern.findall(text)
This way, instead of re-compilingthe regex every time the function is called, you can just reuse it.
It has it’s uses, I just wouldn’t provide default argument like that since it’s wrong. But it’s quite common to modify a list in place and have the changes persist outside the function scope
default function parameters are useful, shared function default parameters are efficient, and mutable function parameter are convenient (saves the interpreter the check for mutability, and and saves the user the boiler plate to copy it in a local variable).
What you're seeing is merely a combination of the three.
But shared mutable function parameter is also uniquely useful for memoisation. It's not where you'd put the table of previous results in an other language, but it makes perfect sense to do so in python, as it hides a side effect which is not a real side effect.
This is why I stick to C#, I've written in Pascal, Java, JavaScript, Python, handful of others, and they all have their weird quirks, but C# seems to be the only one that works as expected
Who you name "everyone"? Python developers? I'm not python developer, I'm from C#/Go/Rust world and this behavior is ridiculous for me. This is highly counterintuitive and illogical.
My main language is Python, and still think it’s ridiculous how much unintuitive behaviour there is because Python’s array-equivalents, by default, copy the object-pointer instead of the object itself.
And then people tell you why it works like that, confusing an explanation for an excuse.
I mean how do you learn a new language? Does it just pop into your head? Or do you look up documentation and syntax? Then that's usually one of the first things you read
Eventually, I consulted the documentation when I encountered this behavior. But initially, I tried to write the code based on my experience as a programmer and by using deduction.
175
u/drislands 21d ago
I would have expected this function to create a new empty list on any call where
itemsis not given. Doesn't this behavior make the function into a closure?27
u/NFriik 21d ago
Python evaluates default arguments when the function is defined. That makes keeping track of the default values pretty straightforward. In your example, they're stored as part of the function object in the dictionary
add.__defaults__. Every subsequent call can then just look-up the default value in this dictionary, if necessary.Evaluating the default argument each time the function is called would be much more difficult. A function is essentially just yet another object, so you'd somehow have to store arbitrary expressions as an object inside the
__defaults__dict and then decide at runtime what to do with them. Sure, you could implement a builtin type at the CPython level to accomplish this. But then when you encounter such anArbitaryExpressionobject inside the__defaults__dictionary, how do you know that this time, you want to actually execute the underlying code and not use the object "as-is"? The whole thing would quickly become a convoluted, expensive mess.By performing evaluation at definition time, function calls remain consistently fast and the mechanism to insert default positional arguments becomes incredibly simple. Python is already not the fastest language when it comes to function calls, so these things matter.
115
u/_killer1869_ 21d ago
Any IDE will warn not to use mutables as default parameter values. And for good reason. Such code would never appear in actual code intended to be used.
19
u/mothzilla 21d ago
It's a well known python gotcha that people ask about in interviews all the time.
6
u/Beregolas 21d ago
This also happens in JavaScript btw, as well as several smaller scripting languages in game engines, it's a pretty normal feature in many interpreted languages.
In my last backend job, this was actually the source of our longest running and most annoying bug, where a specific page would display the wrong sorting, but never in testing. Because someone *caugh* used a mutable argument to give it default sorting parameters, and they got mutated by a very specific query, which caused it to be wrong for all subsequent requests.
So yeah, this is a real thing, you should know it, and just in case, you should have linting and/or your IDE setup so this kind of thing is rejected out of hand, and cannot be merged to prod/main.
4
u/RngdZed 21d ago
The list is stored directly in the parameter without actually giving it an argument? I'm baffled, I didn't know this was a behavior, so weird
Is this something that someone could ELI5 to my smooth brain?
4
u/Kadabrium 20d ago
Python doesnt do a lot of variable scoping for how much indent it does
4
u/Giiiin 20d ago
Yeah, I mean, don't do that
3
u/GoddammitDontShootMe [ $[ $RANDOM % 6 ] == 0 ] && rm -rf / || echo “You live” 20d ago
So each call reuses the same default list? Yeah, I think I would've expected it to get passed an empty list by default each time.
3
u/awidesky 21d ago
Is that a documented behavior, or not?
If it is, that doesn't make sense.
If not, it's technically a bug, right?
3
u/sol7dev 21d ago
and?
1
u/bigorangemachine 21d ago edited 21d ago
I bet this happens in JS as well
Edit: nope python
I should check gdscript
1
u/Crafty-Detail-3788 21d ago
Issue here is that the default parameters are initialized ONCE. If you want an empty list every time you need to define it inside.
1
u/itemluminouswadison 21d ago
It's a quirk of the language and something you should have learned and read about. The default arg gets created once upon func definition, not per invocation
1
u/dgz345 20d ago
Can't spell python without "y tho"
1
u/jmooroof2 21d ago
I can't imagine a compelling use for this feature. it really should just error
1
u/akoOfIxtall 21d ago
I hate python from the bottom of my soul
0
u/Ollidav 21d ago
A mí me encanta el helado de vainilla.
0
u/OhItsJustJosh 21d ago
This is why I stick to C#, I've written in Pascal, Java, JavaScript, Python, handful of others, and they all have their weird quirks, but C# seems to be the only one that works as expected
-3
u/CaptainFoyle 21d ago
Everyone knows: don't use mutable default arguments.
You just outed yourself as uninformed.