Generators and Iterators a la Python

[petermsiegel]

Anyone thinking about Python-like generators and iterators; generators can be thought of as "lazy" lists of mappings (simple or complex) of possibly infinite length.

The key innovation needed would be for APL built-in functions and operators, when faced with a user-defined class object, to expect and call certain class members, e.g. myclass.next, as well as to respond to certain signals (modeling Python StopIteration). While there are many elegant uses of generators and iterators we just don't need in APL, there are some uses already in place in an ad hoc fashion -- note how selection works today with certain default class methods (where it lazily waits before calling the method until it knows which object components it needs).

The <yield> keyword in Python (which allows generators to be programmed as if regular (co-)functions), yielding a value, while maintaining the current state, rather then returning a value. It can be simulated using threads, but requires different user functions to stay out of each other's way (e.g. in token management or using local variables to synchronize).

See:http://www.python.org/dev/peps/pep-0255/ (also, less elegantly, in many other languages).

Seems easy enough! Devils in the details.

[JohnS|Dyalog]

While nowhere near as neat as primitive language constructs, some of this functionality may already be modelled using namespaces, bound as operands or left arguments, to maintain persistent local state. See:
Function memoization: http://dfns.dyalog.com/n_memo.htm
Editor undo/redo stacks: http://dfns.dyalog.com/n_UndoRedo.htm
and this experimental version of dyalog, which implemented closures:
Closures: http://dfns.dyalog.com/downloads/fre.pdf

[petermsiegel]

I had seen the closures article, which was spot on; closures would be just the tool needed. But they aren't part of the product, right?

[JohnS|Dyalog]

Closures are not part of the product. You can model persistent local state by binding a namespace as left argument/operand:

Code: Select all

      next←(⎕ns'')∘{                    ⍝ space bound as left arg.
          0=⍵:⍺.n←0                     ⍝ next 0: reset number stream.
          (⍳⍺.n+←⍵)+⍺.n                 ⍝ next ⍵: return next ⍵ numbers.
      }

      next 0                            ⍝ reset count.

      next 1                            ⍝ next number.
1
      next 1                            ⍝ next number.
2
      next 2                            ⍝ next two numbers.
3 4
      next 1                            ⍝ next number.
5
      next 0                            ⍝ reset count.

      next 3                            ⍝ next three numbers,
1 2 3


See http://dfns.dyalog.com/n_memo.htm for more examples.
John.

[David Lamkins]

That's a neat technique.

Is there a reason I can't fix the 'next' function in Dyalog 13.1 for Linux?

[David Lamkins]

BTW, the technique works for me if I break it down into two functions, like this:

Code: Select all

 next∆←{
     0=⍵:⍺.n←0
     (⍳⍺.n+←⍵)+⍺.n
 }
 next←(⎕ns'')∘next∆                                                                             


It seems that my Linux version of Dyalog can't parse composition as part of a d-function definition.

[Vince|Dyalog]

Hi David,

Our session doesn't allow you to join together multiple lines into one action. But, you can get this derived function next if you use diamonds and type it on one line in the session like this:

Code: Select all

next←(⎕ns'')∘{ 0=⍵:⍺.n←0 ⋄ (⍳⍺.n+←⍵)+⍺.n}


Regards,

Vince

[JohnS|Dyalog]

Apologies, I forgot that I was using a short-hand. Vince's one-liner-with-diamonds is good for short functions and your 2-stage definition is better for larger ones.

Note that you can also define next as an operator, which leaves the left argument of the resulting derived function free.

Code: Select all

      next∆←{⍺←0          ⍝ default: no reset:
         ⍺:⍺⍺.n←⍵-⍳1      ⍝ reset sequence to ⍵.
         (⍳⍺⍺.n+←⍵)+⍺⍺.n  ⍝ next ⍵ numbers.
      }

      next ← (⎕ns'')next∆  ⍝ NB: no '∘'

      1 next 0   ⍝ reset sequence to 0.

      next 2     ⍝ next 2 numbers
0 1
      next 0     ⍝ next 0 numbers

      next 3     ⍝ next 3 numbers
2 3 4
      1 next 42  ⍝ reset sequence to 42

      next 2     ⍝ next 2 numbers
42 43


NB: In the above, the multi-line D-fn may not be entered directly into the session.
Use )ed next∆ to define it.

[David Lamkins]

Thank you, John and Vince.

Your replies had me scratching my head for a moment. I know that dfns must be defined on one line in the session. I tried that this morning; of course it works.

Yesterday, though, I was attempting to define 'next' in an editor. This (your original version using the namespace composed with the function) gives me a "Can't Fix" message in the mode line regardless of whether the definition is on one line or not.

It seems, therefore, that the editor and the session have different syntax requirements for dfns. Is this expected?

[JohnS|Dyalog]

It seems, therefore, that the editor and the session have different syntax requirements for dfns. Is this expected?

Yes, in this case, the session is being used to name the result of the evaluation of an expression:

      half ← ÷2              ⍝ naming result of function applied to array argument
      sum  ← +/              ⍝ naming result of operator applied to function operand
      next ← (⎕ns'')∘{...}   ⍝ naming result of operator applied to operands

The editor can define only the components (arrays, functions, operators) of such expressions.

John

PS. great advances have often arisen from questions such as these.