/ 
Zoidberg-0.981
River stage zero No dependents
/ zoiddevel

NAME

zoiddevel - Development documentation for zoid

DESCRIPTION

Debugging

If you switch on the global debug bit and the verbose bit both your input and debug information are echoed to STDERR. Thus by piping both STDOUT and STDERR to a file you get a detailed log of what is happening.

To set both bits type:

zoid$ set debug
zoid$ set verbose

or start zoid with:

$ zoid -vD

You can also get debug information from just one module by setting a global variable $DEBUG in the module's namespace.

If you set the debug variable to a number higher than 1, you get a stack trace of that number of frames for each exception thrown using the error() method.

Structure

class diagram

Zoidberg::Shell
      /|\            Zoidberg::Contractor
       |                 /|\   /|\
       |__________________|     |
            |                   |
            |            Zoidberg::Job
         Zoidberg              /|\
                                |
                                |
                         Zoidberg::Job::Builtin

All other classes stand on their own (except perhaps for some internal helper classes).

Zoidberg::Shell is an interface class to be used in perl scripts like zoidrc. Zoidberg inheriting from it in order to make this interface also available through the $shell object in the eval scope and to plugins. The Zoidberg::Contractor class contains the code to manage Jobs. Because the Job class also inherits from it sub-jobs are possible.

object structure

main object  class Zoidberg
  |
  |___ {eval}          Zoidberg::Eval object
  |___ {stringparser}  Zoidberg::StringParser object
  |
  |___ {jobs}          Array with Job objects
  |      |_ items of class Zoidberg::Job
  |
  |___ {settings}      Settings and config hashes
  |___ {objects}       Plugin objects       - tied Zoidberg::PluginHash
  |___ {parser}        Custom parsing       - tied Zoidberg::DispatchTable
  |___ {events}        Event code refs      - tied Zoidberg::DispatchTable
  |___ {commands}      Command code refs    - tied Zoidberg::DispatchTable
  |___ {aliases}       Alias definitions

FIXME explanation

Process flow

     start
       |
       |      incomplete input - readmore
       |    .--------------------.
       |    |                    |
       |    |                    |
       V    V   string           |           tree
 input loop     -------->  parse string      ------>  execute tree
 Z:main_loop()             Z:shell_string()           Z:C:shell_list()
 readline                                               |
       A                                   reincarnate  |
       |                                       ,------> |
new    |                           next chunk  |        |
input  |                           of blocks   |        | logic chunk
       |                                       |        |
       |                                       |        V
       '---------------------------------------+---  execute job
                                                     Z:J:exec()
                                                        
Z:   = Zoidberg::
Z:C: = Zoidberg::Contractor::
Z:J: = Zoidberg::Job::

The parse tree going to shell_list() is a simple parse tree consisting of scalars and scalar references (see FIXME below). This tree is processed in logic chunks, corresponding to one job each. The blocks references in a logic chunk are parsed by parse_block() just before executing. This is where the context is determined and things like expansions and subtitutions are done.

Parsing

FIXME put devel opts in scheme

string  --> shell_string()

          split script_gram
                 
                |
                V

list    --> shell_list()   ... FIXME tell bout this format below
       
                |
                V
 
 ref   -->  parse_block()   ... FIXME tell bout possible block forms
                
                |
                V                 ------------.
                                              |
           parse_env()                        | if word block
                                              |
           split word_gram                    |
           split redir_gram                   |
         strip env declaration     A          |
                                   |          |
               |           <------------------'
               V                   |
         alias expansion    -------' if multiple blocks result
         custom filtering
         decide context unless already known
         - context on whole string like perl
         - custom word lists
         - default word context
             
               |----> custom parser
               |
               |----> parse_perl()  for perl context
               |
               | for word context   ---------.  if 'FIXME'
               V                             |
                                             |
           parse_words()                     V

           expansion   ... FIXME list them in zoiduser
           (globs)

A block that originates from shell() or Zoidberg::Shell::AUTOLOAD is recognised as having already been split into words. This means the parsing of words, redirections and environment declarations is skipped. Only alias are checked so this interface is consistent in command names etc. with the commandline interface.

If you use builtin() or command(), these are understood as simple command with parsed words and effectively all parsing is skipped.

Parse tree

FIXME tell about the blocks used in lots of hooks

FIXME renice this stuff

This part describes the form of a parse tree as used between the various Zoidberg objects.

Example

# Commandline input:

$ ls -al | perl{ while (<STDIN>) { print $_ }  }abc && echo done

# Would be parsed to:

[ 
  [{context => 'CMD'}, qw/ls -al/],
  [{context => 'PERL', opts => 'abc'}, q{ while (<STDIN>) { print $_ } } ],
  'AND',
  [{context => 'CMD'}, qw/echo done/]
]



# Commandline input:

$ cd .. && ls -al ; cp dus ~/tmp/ &

# Would be parsed to:

[
  [{context => 'CMD'}, qw/cd ../],
  'AND',
  [{context => 'CMD'}, qw/ls -al/],
  'EOS',    # End Of Statement
  [{context => 'CMD'}, qw{cp dus ~/tmp/}],
  'EOS_BG'  # background statement
]
 
# FIXME an example with redirections

There are a lot more meta fields like 'fork_job', FIXME where to read bout them?

Basics

A parse tree is an array consisting of blocks and tokens. A block can be any kind of shell code and is stored in a nested array. Blocks directly following each other are supposed to be a pipeline. A token is a delimiter between blocks.

The first field of a block is a hash which contains information about the block, all other field in a block make up the content. The most important information about a block is the context, which tells the parser how to execute the block. You are free to store all kinds of specific information in this first field, but some key names are reserved.

FIXME reserved meta fields

Pseudo parse trees

These are forms that can be used with the shell() function provided by Zoidberg::Shell. Just as by the real parse tree blocks of code are references and tokens are plain scalars. A block that is a scalar reference will be split into words and parsed completely (although still expected to be one block). A block that is an array reference will be considered to be completely parsed if the first element is a hash reference and the context defined, else it is considered a word list, possibly with meta data defined in the first element.

# for example "ls -al | perl{ while (<STDIN>) { print $_ }  }abc && echo done"
# can be executed by calling :
shell(
  [qw/ls -al/],
  \'perl{ while (<STDIN>) { print $_ }  }abc',
  'AND'
  [{context => 'CMD'}, qw/echo done/]
);

Using this kind of pseudo trees only makes sense if you are lazy or you don't know exactly what the command is but you have some clues you want to pass to the parser.

Built-in contexts

The contexts 'PERL', 'CMD' and 'SUBZ' are used by the built-in parser: 'PERL' flags code as perl code, 'CMD' as a system or built-in command and the 'SUBZ' context is used for sub-shells.

Parser hooks

The hash called 'parser' in the main object is a tied hash of the class Zoidberg::DispatchTable. This hash contains stacks of code refs to be called at certain parser stages.

Subroutines in these stacks should in general return a BLOCK (a array ref with a meta field and words) at success and undef otherwise.

filter BLOCK

Stack called before contexts are decided and before the block is splitted into words. Used to claim a block for a certain context by setting the 'context' field in the meta hash.

word_expansion BLOCK

Allows you to define a custom expansion routine.

word_list BLOCK

Stack with a dual function: used to claim a block for a certain context after word splitting, but also for listing completions for this context.

If called in wantarray context should return a list of possible completions for the first word in BLOCK.

If not in wantarray context can either return a block with the 'context' field set to claim this block for a certain context.

You can define a custom context by putting a sub-hash into the parser hash (using the context name as the key). See "Custom contexts" for the keys to be defined in such a sub-hash (note that some of the stacks are included there also - but these won't go in the sub-hash).

Settings

Notice that every plugin has it's own config hash in the settings hash, it is suggested that settings only affecting one plugin are placed in this sub-hash. This rule is broken for some settings required by the posix specification of the set built-in.

FIXME list these posix settings somwhere

Below are listed advanced settings only needed for development. Common settings can be found in zoiduser.

The parser settings can be set globally, but also per block. FIXME they should also be settable per mode/context

split_script

Split a command according to the script grammar.

split_words

Split a command into words.

parse_fd

Parse redirections.

parse_env

Parse environment declerations.

parse_aliases

Check for aliases.

parse_def_contexts

Use the default contexts PERL and CMD.

expand_comm

Expand commands in "word" contexts.

expand_param

Expand parameters in "word" contexts.

expand_path

Expand globs in "word" contexts.

debug

Turn on _all_ debug messages.

plain_words

This is not a global setting, but if used for a single block it is identical to turning off all parser settings except for 'parse_aliases'.

Events

The hash called 'events' in the main object is a tied hash of the class Zoidberg::DispatchTable. This hash contains stacks of code refs to be called when certain events happen.

Some events used:

FIXME more events

beat

Called every idle second when using Term::ReadLine::Zoid. Used for asynchronous stuff.

newpwd

Called if $ENV{PWD} has changed. The old pwd can be found in $ENV{OLDPWD}.

prompt

This event is called from the main loop just before respawning the prompt. It can be used to update routines.

cmd

This event is called after the input prompt with the commandline as the first argument. It is intended for interactive input only.

envupdate

This event is called after the spawning of every job. Like 'prompt' it can be used for update routines; since 'envupdate' is called more often then 'prompt', use 'prompt' if it makes no difference, 'envupdate' should only be used for atomic updates.

loadrc

Called after all the plugins are loaded, it triggers reading the rcfiles. Can be used by plugins to set hooks that depend on other plugins being loaded.

exit

Called just before round_up(), used as replacement for .bash_logout.

set_*

Whenever a setting is changed a event is called which is the name of the setting prefixed by 'set_' . Arguments are the new value and the old value of the setting. When the new value is 'undef' the setting was deleted.

These events are generated by the Zoidberg::SettingHash class, which is used to tie the settings hash of the main Zoidberg object.

plug_*
unplug_*

Called when a plugin is loaded or deleted.

Plugins

Zoid makes extensive use of plugins to perform tasks that are not considered the core functionality of a shell (and very little is). A plugin has three ways of communicating with the the shell and other plugins:

builtins

These are commands implemented in the shell to offer some function to both the user and other plugins.

events

A plugin can both trigger events ('broadcast') or listen to certain events. When an event is triggered all code 'listening' to it is executed, this is a one-to-many dispatcher.

contexts

Used to extend the code types zoid knows. By default only command syntax and perl are recognized, but a plugin might want to add other syntaxes.

A context is typically used when it is unconvenient to define all commands the plugin can handle as builtins, or when a plugin want to handle the whole code block instead of the individual (possibly expanded) words.

Also some methods are provided to hook into certain parser stages.

Some general tips for writing plugins:

  • See the Zoidberg::Utils namespace for common functions. Using these functions will make your plugin more coherent with the rest of the shell.

  • Try to use api's, not data structures; unless the structures are specified in the documentation they can change without notice. Api's are in general more stable.

  • Try to use AutoLoader if you have a lot of optional functionality.

    ( When using AutoLoader make sure your subs have names that are case-insensitive unique, else you'll run into problems on OS X. )

  • When you need hooks between two plugins, use a builtin command. Make sure to find a name for it that doesn't overload a normal system command. When you need various hooks, see if they can be combined in one builtin command using options (see Zoidberg::Utils::GetOpt). Also check if other shell implementations (like bash, tcsh or zsh) have builtins for similar funtions and try to make your interface consistent.

    When calling a builtin as a hook the most efficient way is to use the $shell->builtin($cmd, @args) method in a list context. Using this method will cause a lot of parser levels to be skipped, also the command won't be subject to things like the current command mode. Using list context makes the method return directly without bothering with overloaded objects.

PluginConf

A plugin configuration file can use the following keys:

module

The class to bless the plugin object in.

config

Hash with plugin specific config stuff. For plugins that inherit from Zoidberg::Fish this will automatically become the {config} attribute.

commands

Hash linking command names to subroutines. See Zoidberg::DispatchTable for the string format used in this hash.

export

Array with commands automatically linking to a subroutine of the same name in the plugin object.

events

Like commands but used for events.

import

Like export but used for events.

settings

Supply some default settings. Used for global settings, plugin settings should be in the 'config' hash.

aliases

Supply some default aliases.

context

Add context hooks.

If you want your plugin to be loaded as soon as possible, you can use the loadrc event to load your plugin. For plugins inheriting from Zoidberg::Fish loadrc => 'plug' will work.

Custom contexts

Typically you'll want to set a custom context from a plugin. To do this you either provide some context hooks in the PluginConf or you set a loadrc event to load your plugin and then setup the context parser and handlers routines from the plugin's init() routine, using the add_context() method in Zoidberg::Fish. If you defined a word_list or filter hook it makes sense to use the second method. The plugin would be loaded anyway when the first command is parsed.

A context config hash can contain the following routines:

FIXME more verbose description

word_list BLOCK

If you want your context to work with words you should have this hook to recognize and list commands. On wantarray it should return a list of possible completions for the word in $$block[1]; else it should check whether the word is a recognized command and return true on success. In both cases it is also allowed to return a block ref, this is for the more advanced options.

parser BLOCK

Here you can set options like for example fork_job or no_words for your context.

handler BLOCK

This is the part where your command gets executed.

completion_function WORD, LINE, START, END

Hook used by the Intel plugin for custom completion.

intel BLOCK

Hook used by the Intel plugin for custom completion.

filter BLOCK

Hook to filter blocks before they are parsed, this can be used for example for custom redirection code.

Be aware that the a block might be filtered twice with the same meta hash, the second time should reset any meta fields set by the first time.

word_expansion BLOCK

Hook extra expansions into the parser.

"Hello World" plugin

Here follows an example to create a simple plugin with the "hello_world" builtin.

First create a dir ~/.zoid/plugins/HelloWorld.

Next create a module, for example :

package HelloWorldZoid;

# Zoidberg::Fish is the base class for plugins
use Zoidberg::Fish;
our @ISA = qw/Zoidberg::Fish/;

# Zoidberg::Utils provides the output method
use Zoidberg::Utils;

# no need for a constructor, bootstrap with init
sub init { $$_[0]{config}{string} ||= "Hello world !" }

# and in this sub we actually print the string
sub hello_world { output $$_[0]{config}{string} }

# this will be called when the plugin is unloaded
sub round_up { output "someone help me" }

1; # keep require happy

Save the module as ~/.zoid/plugins/HelloWorld/HelloWorldFish.pm.

Then create a config file, this is just a perl script returning a config hash.

{
  module => q/HelloWorldFish/,
  config => {
    # This hash will be both accessible as $shell->{settings}{HelloWorld}
    # and as $shell->HelloWorld->{config}
    string => q/Hello cruel world !/
  },
  export => ['hello_world'],
}

Save the config file as ~/.zoid/plugins/HelloWorld/PluginConf.pl.

After restarting zoid, you should have a builtin command "hello_world" that prints the string "Hello cruel world !", and an object called "HelloWorld". Of course you guessed already that you can control the string that will be printed from the config file, and also that it defaults to "Hello world !". On run-time this string is available under $shell->{settings}{HelloWorld}{string}, it can be changed at any time.

ENVIRONMENT

The following environment variables are used be Zoidberg.

ZOIDPID

Contains the process id of a parent zoidberg shell, intended to be used for an IPC mechanism.

ZOIDREF

In forked child processes this variable contains a stringyfied version of the current Zoidberg object in charge. The parent process has a global hash %Zoidberg::OBJECTS which maps these strings to the original references. It is intended that an IPC mechanism uses this hash to convert strings back to references.

This value should correspond to the object stored in the global $Zoidberg::CURRENT at the time of forking.

In secondary scripts it is better to use Zoidberg::Shell->current().

TODO this mechanism seems to have some problems

ZOIDCMD

Contains the string that caused the current command to be executed. Can be used by a command to inspect its arguments as they were before path expansions etc..

SEE ALSO

perl(1), http://github.com/jberger/Zoidberg