How Mush works

Note

This documentation explains how Mush works using fairly abstract examples. If you’d prefer more “real world” examples please see the Example Usage documentation.

Constructing runners

Mush works by assembling a number of callables into a Runner:

from mush import Runner

def func1():
    print('func1')

def func2():
    print('func2')

runner = Runner(func1, func2)

Once assembled, a runner can be called any number of times. Each time it is called, it will call each of its callables in turn:

>>> runner()
func1
func2

More callables can be added to a runner:

def func3():
    print('func3')

runner.add(func3)

If you want to add several callables in one go, you can use the runner’s extend() method:

def func4():
    print('func4')

def func5():
    print('func5')

runner.extend(func4, func5)

Now, when called, the runner will call all five functions:

>>> runner()
func1
func2
func3
func4
func5

Runners can also be added together to create a new runner:

runner1 = Runner(func1)
runner2 = Runner(func2)
runner3 = runner1 + runner2

This addition does not modify the existing runners, but does give the result you’d expect:

>>> runner1()
func1
>>> runner2()
func2
>>> runner3()
func1
func2

This can also be done by passing runners in when creating a new runner or calling the extend method on a runner, for example:

runner1 = Runner(func1)
runner2 = Runner(func2)
runner4_1 = Runner(runner1, runner2)
runner4_2 = Runner()
runner4_2.extend(runner1, runner2)

In both cases, the results are as you would expect:

>>> runner4_1()
func1
func2
>>> runner4_2()
func1
func2

Finally, runners can be cloned, providing a way to encapsulate commonly used base runners that can then be extended for each specific use case:

runner5 = runner3.clone()
runner5.add(func4)

The existing runner is not modified, while the new runner behaves as expected:

>>> runner3()
func1
func2
>>> runner5()
func1
func2
func4

Configuring Resources

Where Mush becomes useful is when the callables in a runner either produce or require objects of a certain type. Given the right configuration, Mush will wire these together enabling you to write easily testable and reusable callables that encapsulate specific pieces of functionality. This configuration is done either imperatively, declaratively or using a combination of the two styles as described in the sections below.

For the examples, we’ll assume we have three types of resources:

class Apple:
    def __str__(self):
        return 'an apple'
    __repr__ = __str__

class Orange:
    def __str__(self):
        return 'an orange'
    __repr__ = __str__

class Juice:
    def __str__(self):
        return 'a refreshing fruit beverage'
    __repr__ = __str__

Specifying requirements

When callables take parameters, Mush can be configured to pass objects of the correct type that have been returned from previous callables in the runner. For example, consider the following functions:

def apple_tree():
     print('I made an apple')
     return Apple()

def magician(fruit):
     print('I turned {0} into an orange'.format(fruit))
     return Orange()

def juicer(fruit1, fruit2):
     print('I made juice out of {0} and {1}'.format(fruit1, fruit2))

The requirements are specified by passing the required type in the requires parameter when adding the callable to the runner using add(). If more complex requirements need to be specified, a requires instance can be passed which accepts both positional and keyword parameters that specify the types required by the callable being added:

from mush import Runner, requires

runner = Runner()
runner.add(apple_tree)
runner.add(magician, requires=Apple)
runner.add(juicer, requires(Apple, fruit2=Orange))

Calling this runner will now manage the resources, collecting them and passing them in as configured:

>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange

Optional requirements

It may be that, while a callable needs an object of a particular type, a default can be used if no such object is present. Runners can be configured to take this into account. Take the following function:

def greet(name='stranger'):
    print('Hello ' + name + '!')

If a name is not always be available, it can be added to a runner as follows:

from mush import Runner, optional

runner = Runner()
runner.add(greet, requires=optional(str))

Now, when this runner is called, the default will be used:

>>> runner()
Hello stranger!

The same callable can be added to a runner where the required strings is available:

from mush import Runner, optional

def my_name_is():
   return 'Slim Shady'

runner = Runner(my_name_is)
runner.add(greet, requires=optional(str))

In this case, the string returned will be used:

>>> runner()
Hello Slim Shady!

Using parts of a resource

Resources can have attributes or items that are directly required by callables. For example, consider these two functions that return such resources:

class Stuff(object):
    fruit = 'apple'
    tree = dict(fruit='pear')

def some_attributes():
    return Stuff()

def some_items():
    return dict(fruit='orange')

Also consider this function:

def pick(fruit1, fruit2, fruit3):
    print('I picked {0}, {1} and {2}'.format(fruit1, fruit2, fruit3))

All three can be added to a runner such that mush will pass the correct parts of the returns resources through to the pick() function:

from mush import Runner, attr, item, requires

runner = Runner(some_attributes, some_items)
runner.add(pick, requires(fruit1=attr(Stuff, 'fruit'),
                          fruit2=item(dict, 'fruit'),
                          fruit3=item(attr(Stuff, 'tree'), 'fruit')))

So now we can pick fruit from some interesting places:

>>> runner()
I picked apple, orange and pear

The pick() function, however, remains usable and testable on its own:

>>> pick('apple', 'orange', 'pear')
I picked apple, orange and pear

Specifying returned resources

As seen above, Mush will track resources returned by callables based on the type of any object returned. This is usually what you want, but in some cases you may want to specify something different. For example, if you have a callable that returns a sequence of resources, this would be added to a runner as follows:

from mush import Runner, returns_sequence

def all_fruit():
    print('I made fruit')
    return Apple(), Orange()


runner = Runner()
runner.add(all_fruit, returns=returns_sequence())
runner.add(juicer, requires(Apple, Orange))

Now, the juicer will use all the fruit returned:

>>> runner()
I made fruit
I made juice out of an apple and an orange

In rarer circumstances, you may need to override the types returned by a callable. This can be done as follows:

from mush import Runner, returns

class Tomato:
    def __str__(self):
        return 'a tomato'

class Cucumber:
    def __str__(self):
        return 'a cucumber'

def vegetables():
    print('I made vegetables')
    return Tomato(), Cucumber()

runner = Runner()
runner.add(vegetables, returns=returns(Apple, Orange))
runner.add(juicer, requires(Apple, Orange))

Now, even when a callable requires fruit, we can force it to be happy with vegetables:

>>> runner()
I made vegetables
I made juice out of a tomato and a cucumber

The returns indicator can be used even if a single object is returned.

Another way that the type used to track the resource can be different from the type of the resource itself is if a callable returns a mapping and Mush is configured to use the types from that mapping:

from mush import Runner, returns_mapping

def desperation():
    print('I sold vegetables as fruit')
    return {Apple: Tomato(), Orange: Cucumber()}

runner = Runner()
runner.add(desperation, returns=returns_mapping())
runner.add(juicer, requires(Apple, Orange))

Once again, we can happily make juice out of vegetables:

>>> runner()
I sold vegetables as fruit
I made juice out of a tomato and a cucumber

Finally, if you have a callable that returns results that you wish to ignore, you can do so using nothing:

from mush import Runner, nothing

def spam():
    return 'spam'

runner = Runner()
runner.add(spam, returns=nothing)

Named resources

Sometimes the types of resources are too common for them to uniquely identify a resource:

def age():
    return 37

def meaning():
    return 42

A callable such as the following cannot be configured to require the correct resource from these two functions by type alone:

def profound(age, it):
    print('by the age of %s I realised the meaning of life was %s' % (
        age, it
    ))

For these situations, Mush supports the ability to name a resource using a string:

runner = Runner()
runner.add(age, returns='age')
runner.add(meaning, returns='meaning')
runner.add(profound, requires('age', it='meaning'))

Anywhere that a type can be used, a string name can be used instead:

>>> runner()
by the age of 37 I realised the meaning of life was 42

Using Type Annotations

While the imperative configuration used so far means that callables do not need to be modified, Python’s type annotations can also be used to specify requirements and returned resources:

from mush import requires

def apple_tree():
    print('I made an apple')
    return Apple()

def magician(fruit: Apple) -> 'citrus':
    print('I turned {0} into an orange'.format(fruit))
    return Orange()

def juicer(fruit1: Apple, fruit2: 'citrus'):
    print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
    return Juice()

These can now be combined into a runner and executed. The runner will extract the requirements from the type annotations and will use them to map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

Declarative configuration

When type annotations are not available, either because they’re being used for something else or because of the version of Python being used, the helpers for specifying requirements and return types can also be used as decorators:

from mush import requires

def apple_tree():
    print('I made an apple')
    return Apple()

@requires(Apple)
@returns('citrus')
def magician(fruit):
    print('I turned {0} into an orange'.format(fruit))
    return Orange()

@requires(fruit1=Apple, fruit2='citrus')
def juicer(fruit1, fruit2):
    print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
    return Juice()

These can now be combined into a runner and executed. The runner will extract the requirements stored by the decorator and will use them to map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

Default configuration

If no declarations are made using either decorators or type annotations, then arguments that are needed by a callable will be looked up based on the name of the argument:

from mush import requires

def apple_tree() -> 'apple':
    print('I made an apple')
    return Apple()

def magician(apple) -> 'citrus':
    print('I turned {0} into an orange'.format(apple))
    return Orange()

def juicer(apple, citrus):
    print('I made juice out of {0} and {1}'.format(apple, citrus))
    return Juice()

These can now be combined into a runner and executed. The runner will guess the requirements base on the names of the arguments for each function and will use them to map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

If an argument has a default, then the requirement will be made optional.

Configuration Precedence

The four styles of configuration are entirely interchangeable and you can use any combination that suites your requirements.

In terms of precedence, requirements and returned resource specifications will be used in the following order, with the first one found being the one that is used:

• Imperative configuration.

• Declarative configuration.

• Type annotations.

• Default configuration.

The default configuration for requirements is described above.

The default configuration for return values is that a callable’s return value is used as a resource and will be registered against the type of the object returned. The returns_result_type declaration encapsulates this behaviour.

Labels

One of the motivating reasons for Mush to be created was the ability to insert callables at a point in a runner other than the end. This allows abstraction of common sequences of calls without the risks of extracting them into a base class.

The points at which more callables can be inserted are created by specifying a label when adding a callable to the runner. This marks the point at which that callable is included so that it can be retrieved and appended to later. As an example, consider a ring and some things that can be done to it:

class Ring:
    def __str__(self):
        return 'a ring'

def forge():
    return Ring()

def engrave(ring):
    print('engraving {0}'.format(ring))

These might be added to a runner as follows:

from mush import Runner

runner = Runner()
runner.add(forge)
runner.add_label('forged')
runner.add(engrave, requires=Ring)
runner.add_label('engraved')

Now, suppose we want to polish the ring before it’s engraved and then package it up when we’re done:

def polish(ring):
    print('polishing {0}'.format(ring))

def package(ring):
    print('packaging {0}'.format(ring))

We can insert these callables into the runner at the right points as follows:

runner['forged'].add(polish, requires=Ring)
runner.add(package, requires=Ring)

This results in the desired call order:

>>> runner()
polishing a ring
engraving a ring
packaging a ring

Now, suppose we want to polish the ring again after it’s been engraved. We can insert another callable at the appropriate point:

def more_polish(ring):
    print('polishing {0} again'.format(ring))

runner['engraved'].add(more_polish, requires=Ring)

Mush will do the right thing when this runner is called:

>>> runner()
polishing a ring
engraving a ring
polishing a ring again
packaging a ring

When using labels, it’s often good to be able to see exactly what is in a runner, what order it is in and where any labels point. For this reason, the representation of a runner gives all this information:

>>> runner
<Runner>
    <function forge ...> requires() returns_result_type()
    <function polish ...> requires(Ring) returns_result_type() <-- forged
    <function engrave ...> requires(Ring) returns_result_type()
    <function more_polish ...> requires(Ring) returns_result_type() <-- engraved
    <function package at ...> requires(Ring) returns_result_type()
</Runner>

As you can see above, when a callable is inserted at a label, the label moves to that callable. You may wish to keep track of the initial point that was labelled, so Mush supports multiple labels at each point:

>>> runner = Runner()
>>> point = runner.add(forge)
>>> point.add_label('before_polish')
>>> point.add_label('after_polish')
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- after_polish, before_polish
</Runner>

Now, when you add to a specific label, only that label is moved:

>>> point = runner['after_polish']
>>> point.add(polish)
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type() <-- after_polish
</Runner>

Of course, you can still add to the end of the runner:

>>> runner.add(package)
<mush.modifier.Modifier...>
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type() <-- after_polish
    <function package ...> requires('ring') returns_result_type()
</Runner>

However, the point modifier returned by getting a label from a runner will keep on moving the label as more callables are added using it:

>>> point.add(more_polish)
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type()
    <function more_polish ...> requires('ring') returns_result_type() <-- after_polish
    <function package ...> requires('ring') returns_result_type()
</Runner>

Plugs

You may run into situations where you wish to group callables together and add them in one go. Indeed, it might only make sense to add all callables in a group and would cause problems to add them individually.

For example, suppose we have this runner:

def prepare():
    print('cleaning kitchen table')

def wash(produce):
    print('washing '+str(produce))

def finished():
    print('service please!')

kitchen_runner = Runner()
kitchen_runner.add(prepare)
kitchen_runner.add_label('what')
kitchen_runner.add(wash, requires='produce')
kitchen_runner.add_label('how')
kitchen_runner.add(finished)

For any use of the kitchen, we need to specify both what to use and how we want to use it once it’s washed. This can neatly be done as follows:

from mush import Plug

class JuicePlug(Plug):

    def what(self) -> 'produce':
        return Apple()

    def how(self, produce):
        print('juicing '+str(produce))

runner = kitchen_runner.clone()
JuicePlug().add_to(runner)

The runner behaves as we require:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

It may be that we want our plug to have helper methods, in which case they can either be named with a leading underscore, or the plug can be set up to only add explicitly marked methods, for example:

from mush.plug import Plug, insert

class JuicePlug(Plug):

    explicit = True

    def juice(self, produce):
        print('juicing '+str(produce))

    @insert()
    def what(self) -> 'produce':
        return Apple()

    @insert()
    def how(self, produce: 'produce'):
        self.juice(produce)

runner = kitchen_runner.clone()
JuicePlug().add_to(runner)

The runner behaves as before:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

As you can see, this might make for a lot of decorating if you only have one helper method. If that’s the case, you can just tell the plug to ignore the helper:

from mush.plug import Plug, ignore

class JuicePlug(Plug):

    @ignore()
    def juice(self, produce):
        print('juicing '+str(produce))

    def what(self) -> 'produce':
        return Apple()

    def how(self, produce: 'produce'):
        self.juice(produce)

runner = kitchen_runner.clone()
JuicePlug().add_to(runner)

The runner still behaves as before:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

It may be that it makes sense to give your method a name different to the label you wish to add it at. You may also wish to have a plug add a method to the end of the runner where there is no label. Both of these are supported:

from mush.plug import Plug, insert, append

class JuicePlug(Plug):

    @insert(label='what')
    def pick_fruit(self) -> 'produce':
        return Apple()

    def how(self, produce: 'produce'):
        print('juicing '+str(produce))

    @append()
    def relax(self):
        print('...and relax')

runner = kitchen_runner.clone()
JuicePlug().add_to(runner)

The runner now behaves as required:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!
...and relax

Context manager resources

A frequent requirement when writing scripts is to make sure that when unexpected things happen they are logged, transactions are aborted, and other necessary cleanup is done. Mush supports this pattern by allowing context managers to be added as callables:

from mush import Runner, requires

class Transactions(object):

    def __enter__(self):
        print('starting transaction')

    def __exit__(self, type, obj, tb):
        if type:
            print(obj)
            print('aborting transaction')
        else:
            print('committing transaction')
        return True

def a_func():
    print('doing my thing')

def good_func():
    print('I have done my thing')

def bad_func():
    raise Exception("I don't want to do my thing")

The context manager is wrapped around all callables that are called after it:

>>> runner = Runner(Transactions, a_func, good_func)
>>> runner()
starting transaction
doing my thing
I have done my thing
committing transaction

This gives it a chance to clear up when things go wrong:

>>> runner = Runner(Transactions, a_func, bad_func)
>>> runner()
starting transaction
doing my thing
I don't want to do my thing
aborting transaction

Testing

Mush has a couple of features to help with automated testing of runners. For example, if you wanted to test a runner that got configuration by calling a remote web service:

def load_config() -> 'config':
    return json.loads(urllib2.urlopen('...').read())

def do_stuff(username: item('config', 'username'),
             password: item('config', 'password')):
    print('doing stuff as ' + username + ' with '+ password)

runner = Runner(load_config, do_stuff)

When testing this runner, we may want to inject a hard-coded config. This can be done by cloning the original runner and replacing the load_config() callable:

>>> def test_config():
...     return dict(username='test', password='pw')
>>> test_runner = runner.clone()
>>> test_runner.replace(load_config, test_config)
>>> test_runner()
doing stuff as test with pw

If you have a base runner such as this:

from argparse import ArgumentParser, Namespace

def base_args(parser):
    parser.add_argument('config_url')

def parse_args(parser):
    return parser.parse_args()

def load_config():
    return json.loads(urllib2.urlopen('...').read())

def finalise_things():
    print('all done')

base_runner = Runner(ArgumentParser)
base_runner.add(base_args, requires=ArgumentParser, label='args')
base_runner.add(parse_args, requires=ArgumentParser)
point = base_runner.add(load_config, requires=attr(Namespace, 'config_url'),
                        returns='config')
point.add_label('body')
base_runner.add(finalise_things, label='ending')

That runner might be used for a specific script as follows:

def job_args(parser: ArgumentParser):
    parser.add_argument('--colour')

def do_stuff(username: item('config', 'username'),
             colour: attr(Namespace, 'colour')):
    print(username + ' is '+ colour)

runner = base_runner.clone()
runner['args'].add(job_args)
runner['body'].add(do_stuff)

To test this runner, we want to use a dummy configuration and command line and not have any finalisation take place. This can be achieved with a helper function such as the following:

def run_with(source_runner, config, argv):
    runner = Runner(ArgumentParser)
    runner.extend(source_runner.clone(added_using='args'))
    runner.add(lambda parser: parser.parse_args(argv),
               requires=ArgumentParser)
    runner.add(lambda: config, returns='config')
    runner.extend(source_runner.clone(added_using='body'))
    runner()

The helper can then be used as follows:

>>> run_with(runner,
...          config=dict(username='test', password='pw'),
...          argv=['--colour', 'red'])
test is red

Debugging

Mush has a couple of features to aid debugging of runners. The first of which is that the representation of a runner will show everything in it, in the order it will be called and what each callable has been declared as requiring and returning along with where any labels currently point.

For example, consider this runner:

from mush import Runner

def make_config() -> 'config':
    return {'foo': 'bar'}

def connect(foo: item('config', 'foo')):
    return 'connection'

def process(connection):
    print('using ' + repr(connection))

runner = Runner()
point = runner.add(make_config, label='config')
point.add(connect)
runner.add(process)

To see how the configuration panned out, we would look at the repr():

>>> runner
<Runner>
    <function make_config ...> requires() returns('config')
    <function connect ...> requires(foo='config'['foo']) returns_result_type() <-- config
    <function process ...> requires('connection') returns_result_type()
</Runner>

As you can see, there is a problem with this configuration that will be exposed when it is run. To help make sense of these kinds of problems, Mush will add more context when a TypeError or ContextError is raised.