Specifying a relative interpreter in a shebang

Background

In order to make a Python script executable it needs to have a shebang line specifying the interpreter of the script.

$ head myscript.py -n 1
#!/usr/bin/python3

This enables passing the script directly to execve where the kernel can eventually call /usr/bin/python3 with the script file as an argument. This works well but has the drawback of harcoding the interpreter. If a different interpreter is needed, like one located in a virtual environment, it cannot be used.

Using env

Instead of hardcoding an interpreter, the shebang line can use env(1). Passing env an argument cases env to lookup that name in the $PATH and execute that as the interpreter.

$ head myscript.py -n 1
#!/usr/bin/env python3

The user can then move their desired python3 binary to the front of the $PATH and that will be used to execute the script.

Limitations of env

There are two limitations with using env like above. First, it’s not possible to pass arguments to python3 via env because env interprets the additional arguments as the name of the interpreter. For example trying to pass the -E flag to python3 results in the following error.

$ head myscript.py -n 1
#!/usr/bin/env python3 -E
$ ./myscript.py
/usr/bin/env: ‘python3 -E’: No such file or directory

This is because env does not split the arguments by spaces before passing them to execve.

$ strace --trace=execve ./myscript.py
execve("./myscript.py", ["./myscript.py"], 0x7ffde39948c8 /* 49 vars */) = 0
execve("/usr/local/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)
execve("/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)
/usr/bin/env: ‘python3 -E’: No such file or directory

Second, this approach assumes the desired interpreter is on the front of the $PATH. If it’s not possible to put the desired interpreter on the $PATH then the correct interpreter will not be used.

Using env -S

It’s possible to overcome the above limitations by using the -S flag to env. The documentation explains that the -S flag will split arguments on whitespace and pass them all to execve. This means instead of just specifying a binary name, all of the values of execve can be specified, such as specifying -E to the interpreter.

$ head myscript.py -n 1
#!/usr/bin/env -S python3 -E

This can be verified by passing -v to get verbose output from env.

$ head myscript.py -n 1
#!/usr/bin/env -vS python3 -E
$ ./myscript.py
split -S:  ‘python3 -E’
 into:    ‘python3’
     &    ‘-E’
executing: python3
   arg[0]= ‘python3’
   arg[1]= ‘-E’
   arg[2]= ‘./myscript.py’
...

Selecting a relative interpreter

Since the -S flag allows us to pass full arguments to an interpreter, it’s possible then to encode an entire program in the shebang line as an argument to another interpreter and have that execute our desired interpreter.

For example we could use env -S to execute /bin/sh and pass in a full script to -c, where that script locates the desired interpreter relative to the script and then executes that. The documentation explains that spaces can be escaped with \_.

$ head myscript.py -n 1
#!/usr/bin/env -vS\_/bin/sh\_-xc\_"exec\_\$(dirname\_"\$0")/venv/bin/python3\_-E\_"\$0"\_"\$@""$ ./myscript.py
split -S:  ‘\\_/bin/sh\\_-xc\\_"exec\\_\\$(dirname\\_"\\$0")/venv/bin/python3\\_-E\\_"\\$0"\\_"\\$@""’
 into:    ‘/bin/sh’
     &    ‘-xc’
     &    ‘exec $(dirname $0)/venv/bin/python3 -E $0 $@’
executing: /bin/sh
   arg[0]= ‘/bin/sh’
   arg[1]= ‘-xc’
   arg[2]= ‘exec $(dirname $0)/venv/bin/python3 -E $0 $@’
   arg[3]= ‘./myscript.py’
+ dirname ./myscript.py
+ exec ./venv/bin/python3 -E ./myscript.py...

Conclusion

By combining env -S and sh it’s possible to create a shebang line that executes an interpreter relative to the script. This could be used to have a script execute in a specific virtual environment.