Don't let mutability of compound objects fool you!

Posted by Bob on Tue 07 March 2017 in Concepts

In this post I wanted to expand a bit on mutability. As already indicated here and here using mutable default values for methods gets you into trouble.

Last week I had a similar issue with mutability when writing test code for our simple Flask API post:

The only challenge was the isolation of each unit test: I had to do copy the app.items to a backup variable in setUp (a deepcopy to not leave references around) and pass it back in tearDown. ...

It is important to become familiar with shallow vs deep copy when dealing with compound (nested) objects!

The documentation explains it well:

The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances):

  • A shallow copy constructs a new compound object and then (to the extent possible) inserts references into it to the objects found in the original.
  • A deep copy constructs a new compound object and then, recursively, inserts copies into it of the objects found in the original

In our API exercise app.items was a list of dicts, so I needed deepcopy:

def setUp(self):
    self.backup_items = deepcopy(app.items)  
    ...

...

def tearDown(self):
    # reset app.items to initial state
    app.items = self.backup_items

I even ended up adding a test in the test_update method to make sure the backup did not get corrupted:

..
..
self.assertEqual(data['item']['value'], 30)
# proof need for deepcopy in setUp: update app.items should not affect self.backup_items
# this fails when you use shallow copy
self.assertEqual(self.backup_items[2]['value'], 20)  # 20 == org value

See REPL output to clarify this further:

>>> items = [
...     {
...         'id': 1,
...         'name': 'laptop',
...         'value': 1000
...     },
...     {
...         'id': 2,
...         'name': 'chair',
...         'value': 300,
...     },
...     {
...         'id': 3,
...         'name': 'book',
...         'value': 20,
...     },
... ]

>>> items
[{'id': 1, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items2 = items[:]  # shallow copy
>>> items2
[{'id': 1, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items2 == items
True

>>> items[0]['id'] = 5
# oops the items2 copy got updated as well!
# -> id of first item got corrupted (5 != 1)

>>> items2 == items
True

>>> items2  
[{'id': 5, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items
[{'id': 5, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]

# same for copying with list constructor, another shallow copy it turns out
>>> items2 = list(items)
>>> items2 == items
True

>>> items
[{'id': 5, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items2
[{'id': 5, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]

>>> items[0]['id'] = 6

# oops
>>> items2 == items
True

# again both data structures' first item were updated
>>> items
[{'id': 6, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items2
[{'id': 6, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]

# now the right way
>>> from copy import deepcopy
>>> items2 = deepcopy(items)
>>> items == items2
True
>>> items[0]['id'] = 7

# cool
>>> items == items2
False

# backup items2 intact
>>> items
[{'id': 7, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]
>>> items2
[{'id': 6, 'name': 'laptop', 'value': 1000}, {'id': 2, 'name': 'chair', 'value': 300}, {'id': 3, 'name': 'book', 'value': 20}]i

See another example taken from Fluent Python shown on this Reddit thread which inspired me to write this up.

I hope this saves some of you Python developers a debugging headache some day :)


Keep Calm and Code in Python!

-- Bob

>>> next(PyBites)

Get our 'Become a Better Python Developer' cheat sheet sent straight to your inbox by signing up for our Newsletter: