credit

list can be so expressive thanks to features like list comprehension, but sometimes it could be hard to read when compounded with multiple operations in one line. Also it may not be expressive enough for some complex operations. L comes to rescue.

As a second post about fastai2, I would talk about L, one of the foundational base classes heavily used in fastai2. We will highlight 10 features of L with simple examples, and compare it against its equivalent operations in list.

0. Prerequisites

L is packaged in fastcore, a standalone library with all the base classes heavily used in fastai2 (e.g. Transform, ItemTransform … etc). It is a super light and you can easily install it by pip install fastcore.

Alternatively, you can install the whole fastai2 here, but it would be an overkill simply for running through this post.

from fastcore.foundation import L

1. What is L?

So what is L? Similar to list, it is just a container for different objects. You can think of it as a better version of list, with extended functionalities. Those functionalities make L more handy to use especially in data science tasks.

You can define a L object by a list or its elements. When you display (manifested by self.__str__ or self.__repr___) the object, it will show you both the number of elements and its elements:

x = L([1, 3, 4, 5])
y = L(1, 3, 4, 5)
x, y

((#4) [1,3,4,5], (#4) [1,3,4,5])

In terms of display, there is another difference between L and list. For L it shows elements horizontally with limit (i.e. first 10 elements if more than 10 entries) while for list it displays all elements.

x = L([i for i in range(20)])
y = [i for i in range(20)]
x, y

((#20) [0,1,2,3,4,5,6,7,8,9...],
 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

2. L v.s. list

How much better is L compared against list? It is the best to illustrate that with simple examples.

2.1. Splitting list of tuple by L.itemgot

With L.itemgot, you can easily convert a list tuple into separate lists in one line of code. If you are using list, you have to firstly separate the list by zip and then converting each output into list. (originally tuple)

x_l = L([(1, 'apple'), (2, 'orange'), (3, 'banana')])
x_list = [(1, 'apple'), (2, 'orange'), (3, 'banana')]
x, x_list

((#3) [(1, 'apple'),(2, 'orange'),(3, 'banana')],
 [(1, 'apple'), (2, 'orange'), (3, 'banana')])

a, b = x_l.itemgot(0), x_l.itemgot(1)
a, b

((#3) [1,2,3], (#3) ['apple','orange','banana'])

a, b = zip(*x_list)
a, b = list(a), list(b)
a, b

([1, 2, 3], ['apple', 'orange', 'banana'])

Additionally, if the elements are class objects/ dictionaries, you can specify the attributes/ keys using string in itemgot.

x_l = L([{'num': i} for i in range(3)])
x_list = [{'num': i} for i in range(3)]
x_l, x_list

((#3) [{'num': 0},{'num': 1},{'num': 2}], [{'num': 0}, {'num': 1}, {'num': 2}])

x_l.itemgot('num'), [i['num'] for i in x_list]

((#3) [0,1,2], [0, 1, 2])

2.2. Combing lists by L.zipwith

This is the reverse operation of 2.1. In L, concatenating multiple lists into a list of tuple is as easy as calling zipwith. In list, you can implement the same by list comprehension.

a_l, b_l = L(1, 2, 3), L('apple', 'orange', 'banana')
a_l.zipwith(b_l)

(#3) [(1, 'apple'),(2, 'orange'),(3, 'banana')]

a_list, b_list = [1, 2, 3], ['apple', 'orange', 'banana']
[(a, b) for a, b in zip(a_list, b_list)]

[(1, 'apple'), (2, 'orange'), (3, 'banana')]

2.3. Searching indexes by L.argwhere

L.argwhere works like np.argwhere. It outputs the index (indexes) of elements that satisfies your specified criteria. You specify your criteria as a function in the argument f. For list, you can implement the equivalence in list comprehension.

x_l = L([1, 1, 2, 3, 4])
x_list  = [1, 1, 2, 3, 4]
x, x_list

((#3) [(1, 'apple'),(2, 'orange'),(3, 'banana')], [1, 1, 2, 3, 4])

x_l.argwhere(f = lambda i: i > 1)

(#3) [2,3,4]

[i for i in x_list if i > 1]

[2, 3, 4]

2.4. Multiple Indexing by L.__getitem__

There are many ways you can do indexing (i.e. __getitem__) in L. You can index one element by int and you can also index multiple elements by list or L. On the contrary, list does not naturally support indexing of multiple elements.

x_l = L(['apple', 'orange', 'banana'])
x_l[0, 0, 1], x_l[L(0, 0, 1)]

((#3) ['apple','apple','orange'], (#3) ['apple','apple','orange'])

x_list = ['apple', 'orange', 'banana']
[x_list[i] for i in [0, 0, 1]]

['apple', 'apple', 'orange']

2.5. Apply a function element-wise by L.map and L.filter|

L.map and L.filter work similarity to Python prebuilt map and filter. It applies a function f on each of its elements and return a new copy. One difference is that map and filter returns a generator instead of a new copy.

x_l = L(1, 2, 3)
x_l.map(f = lambda i: i**2)

(#3) [1,4,9]

x_l.filter(f = lambda i: i >= 2)

(#2) [2,3]

x_list = [1, 2, 3]
o = map(lambda i: i**2, x_list)
o, [i for i in o]

(<map at 0x15395c95bf50>, [1, 4, 9])

o = filter(lambda i: i >= 2, x_list)
o, [i for i in o]

(<filter at 0x15395c95b450>, [2, 3])

2.6. Create Expressive Key-value Pairs by L.map_dict

L.map_dict is slightly complicated than the above methods. It essentially returns a dict. Its keys are the original elements. You specify a function f to apply on each elements and return as the value of each key. The function f can have multiple arguments. You can specify the additional arguments of f as keyword arguments in L.map_dict.

def name_file_by_idx(i, prefix = None, ext = 'png'):
    if prefix is not None:
        out = f'{prefix}_{i:04}'
    else:
        out = f'{i:04}'
    return f'{out}.{ext}'

x_l = L([i for i in range(4)])
x_l.map_dict(f = name_file_by_idx, prefix = 'class1', ext = 'png')

{0: 'class1_0000.png',
 1: 'class1_0001.png',
 2: 'class1_0002.png',
 3: 'class1_0003.png'}

x_list = [i for i in range(4)]
{k: name_file_by_idx(k, prefix = 'class1', ext = 'png') for k in x_list}

{0: 'class1_0000.png',
 1: 'class1_0001.png',
 2: 'class1_0002.png',
 3: 'class1_0003.png'}

2.7. Chaining Multiple Operations

For the methods I highlighted above, so long as they return another copy of L, you can keep applying another operation right after that. The principle is pretty similar to pandas. As a result, you can chain multiple operations together in just one line of code. I personally find such sequential presentation more readable and intuitive.

x_l = L((0, 'xyz'), (1, 'xyz'), (2, 'xyz'))
x_l.itemgot(0).filter(lambda i: i >= 1).map(lambda i: f'class1_{i:04}.png')

(#2) ['class1_0001.png','class1_0002.png']

x_ls = [(0, 'xyz'), (1, 'xyz'), (2, 'xyz')]
[f'class1_{i[0]:04}.png' for i in x_ls if i[0] >= 1]

['class1_0001.png', 'class1_0002.png']

2.8. Getting Distinct Elements by L.unique

You can easily get distinct elements by L.unique with order respected. For list, you can do the same by firstly converting it to set and then back to list, but the drawback is that the output does not respect order, as shown.

x_l = L(2, 2, 1, 4, 3, 4)
x_l.unique()

(#4) [2,1,4,3]

x_list = [2, 2, 1, 4, 3, 4]
list(set(x_list))

[1, 2, 3, 4]

2.9. Shuffling by L.shuffle

Shuffling is commonly used in data partition. L.shuffle can achieve that by returning a new copy of shuffled list. For list, you need help from random.shuffle to do that. Note that random.shuffle makes change in-place.

x_l = L([i for i in range(5)])
x_l.shuffle()

(#5) [0,3,1,2,4]

import random

x_list = [i for i in range(5)]
random.shuffle(x_list)
x_list

[4, 0, 1, 2, 3]

2.10. Adding None

Adding None to list will return TypeError, so you need to write an additional condition to handle that. But for L, you won’t get any error by adding None.

L(1, 2) + None

(#2) [1,2]

try:
    [1, 2] + None
except TypeError as e:
    print(e)

can only concatenate list (not "NoneType") to list

3. Closing Remarks

Due to limited length of this post, we can’t exhaust all the functionalities of L here, but its design is actually pretty simple. You can refer to its source code to learn more about its functionality!