proposal: ⎕here

[ArrayMacNB]

Being fed-up with the baroque way multi-line nouns have to be created, impatient with the promised enhancements, and frustrated with the ease this can be done in a tradfn, where scripts seemingly make this difficult, I would like to propose a system function be added to Dyalog APL:

r←⎕here delim

so that the result is every line after invocation until the occurrence of the delimiter.

quote ⎕here ')' ⍝ x ≡ ⍎ quote x
Hello
World
)
'Hello' 'World'

This would match behavior in almost all other scripting languages, including ': 0' in J, and '<<' in Perl.

It would allow multiline nested array constants to be used without waiting for an upgrade to the interpreter (think: r←⎕json ⎕nere ')'

Perhaps some application of ⎕src would allow for this?

[Phil Last]

I applaud your joining the throng (?) crying out for something like this but my having spent the last four decades writing stuff that did precisely what you request and the last decade modelling an array notation in APL and trying to persuade Dyalog to implement it; and with Dyalog's finally taking the idea on board and writing their own model, first in APL and soon (next year?) to be implemented in C; you will understand my hope at this stage that Dyalog is not sidetracked into spending time on a half-way-house.

[Adam|Dyalog]

Hi ArrayMacNB,

We agree with you that multi-line strings are a valuable thing.

My array notation proposal from 2017 actually included a proposal that a multi-line string would become a vector of character vectors. However, that had multiple issues:

• If allowed in the session, the user could find themselves in an inexplicably unresponsive state. (Though now with the new multi-line support, this could be mitigated using the same visual indication as for other multi-line constructs, and issuing an interrupt would restore order.)
• It doesn't solve the problem of having to double quotes.
• It becomes a question what to do about leading and trailing whitespace.

There are a few other possibilities to consider. All have to deal with the last question, and each has its own issues.

• A keyword based approach. However, this precludes usage in dfns. It would be awkward if one wanted the :EndXyz in the literal, but this could be mitigated by requiring leading whitespace (which is stripped) before each line, but this means simply pasting a text in between the opening and closing markers won't suffice, and it makes whitespace significant. We could of course limit ourselves to requiring a leading space before a deactivated :End marker, but by now the rules are not simple any more. Another way would be to let the user choose a custom closing indicator that they know won't occur:
  :Text html :Until done
  To end a multi-line string, use:
  :EndText
  :EndText done
• We could add a new string literal using doublequotes (for that sake, this could be done using single quotes too, switching mode by way of spanning multiple lines, much as how the array notation switches mode), one that includes escapes similar to what many other languages have. Having an escape character at the end of the line would allow the string to continue:
  html←"multi-\
  line"
  However, this would require escaping double quotes and every line break
• We could use """ to quote a multi-line string, but that'd be the first ever tri-glyph in APL. And we'd either have to come up with a mechanism for escaping this sequence (e.g. """""""!) or give up and let people use full array notation in such a case.
• Your idea of a system function. This is similar to GNU APL's ⎕INP

But let's just say that the actual syntax wasn't a question. There are still many other important design questions:

• Should the result be a vector of text vectors? In the "almost all other scripting languages" you mention, the result is obvious; a LF-segmented string. This is what you get even in J, with a long article detailing how to convert to other forms. Maybe you want a matrix, or a vector with CRs. Or an OS-dependent delimiter? Or maybe you want one long uninterrupted string, just specified over multiple lines… Maybe the function would need a left argument to choose? Or we need another system function to convert between all these?
• What about whitespace? You may want to indent your content. Should we trim or not? What characters? Only spaces? Tabs too? Other Unicode whitespace characters? Should such trimming be only on left? Configurable? Only/also on the right? And if on the right, (an option to get) a single trailing space? Should it skip empty lines?
• Do we want a way to escape things? The closing string? Control characters?
• Do we want a way to inject APL expressions for execution like JavaScript's backtick strings have `text${expression}text`? GNU-APL allows you to choose opening and closing tags for this. Do we want that?

So, rest assured that we are working on this, but we want to avoid what happened in GNU APL where they first implemented one system for multi-line strings, then realised it was sub-optimal, so they deprecated it and implemented another.

Additional reading material:

• Relevant (long) Wikipedia article
• Answers to this Stack Exchange question

[ArrayMacNB]

The thing about all these 'array notation' proposals is that having ⎕here would make them all user exercises, similar to what

t←⎕json ⎕here ''

would do. Phil Last could do one thing, Dyalog could do another, and all could co-exist.

Sometimes it is easier to solve the bigger problem.

[Adam|Dyalog]

Array notation isn't intended to solve this problem, but to be an integral part of the language, facilitating the writing of arrays rather than their construction.

Again, we agree that some form of multi-line string would be valuable, but we need to carefully consider the design, and must answer all the above-mentioned questions first.

[petermsiegel]

While I like the idea of dealing with HERE text w/o the overhead and gymnastics of scanning function source at run-time, consider a systematic and elegant approach that reflects the existing syntax and style of the language. Avoid monstrosities akin to ⎕OFF, which has an optional RIGHT argument: «⎕OFF 1». Syntactic tricks are not just inelegant, they are a real bane for teaching new users, 'cause they are unmotivated exceptions that require explanation. I won't prove that here-- but APL has a real elegance that one ought to value.

Dyalog APL already has directives (control words) that advise the environment, create side effects, provide the user or system structural information, or control execution. Building on one of Adam's ideas, why not exploit that approach modestly, extended judiciously to dfns.

One idea would be a :HERE directive, somewhat in the style of :IMPLEMENTS, already pretty funky but workable, with some options to handle the 2-3 common result types:

:HERE varname ❮:❮NO❯TRIM❯ ❮:NL | :CR | :STD❯ ❮:ENDS terminating_word❯

where keywords (with the usual colon) could determine if leading blanks are trimmed to the position of the first HERE line (:TRIM vs :NOTRIM, default), whether it returns a vector of char vectors (:STD, the default), or a newline- (:NL) or carriage-return (:CR) terminated string (for HTML usage), etc.

Examples:

Code: Select all

⍝ Assign ted ← '    my stuff' '      more indented stuff'
:HERE ted
    my stuff
       more indented stuff
:ENDHERE

⍝ Set myvar← 'Not the\n  :END OF THE WORLD' (\n denotes the real newline)
   :HERE myvar :TRIM :NL :ENDS terminating_word
           Not the
            :END OF THE WORLD
   :ENDHERE terminating_word


There are some decent alternatives (allow double-quoted strings with embedded newlines), allow statements to be continued over multiple lines, and so on. Each of these has some drawbacks (though I'd prefer them if starting from scratch), especially for interpreted languages trying to sync execution with source code.

My two shekels worth, ⎕OFF 4⊣'NOW'

[Morten|Dyalog]

Nothing to add to the :Here discussion (except perhaps that the name doesn't seem to suggest anything meaningful to me, if anything I would guess it is some kind of OO feature related to ⎕THIS).

I totally agree with the comments about ⎕OFF, it is an abomination. We're considering deprecating it and providing a new ⎕EXIT which would be a regular system function that takes arguments just like anything else. If anyone has suggestions for arguments it should take, other than the obvious exit code, please let us know!

[petermsiegel]

You are right: ":here" is opaque for new users. Here-documents and here-strings are of course the historical baggage of Unix, i.e. well known (and natural) to those (of us) who know them well. Perhaps ":inline" a la Microsoft is more meaningful.

https://en.wikipedia.org/wiki/Here_document

[ArrayMacNB]

> 'here' is opaque for new users.

In the same vein, APL is opaque for new users. The solution, or so I have heard, is education.

Discussion to the propriety of the name distracts. Discussion as to the format of the result (newline delimited v nested) distracts. The 'powers that be co-opted my solution' lament distracts. Feels like the increasing hurricane winds as one approaches the eye.

Given that a tradfn allows this with something like:

aHere←(⎕lc[1] upto target){⍺ 'extract lines' ⎕vr ⍵} CurrentFunction ''

All I would need is something like:

anotherHere←(here upto target) fExtractLines ⎕this.⎕source

Let's not try to solve everything with Version 1. Agile rejected that path decades ago, as APL did much much earlier.

[Morten|Dyalog]

In my opinion, the agile solution would seem to be to implement this in APL according to your own taste - you can write a 2-3 line utility function to do this quite easily. Given that many people have already done this, I don't feel there would be sufficient benefit to users for us to add a "quick hack" to the APL interpreter. It only makes sense to me to add this as a language feature in the hope that everyone might eventually use the same mechanism, IF we take the time to design it well.

Most of the people who use Dyalog APL for research or business tell us that they are very unhappy when we make changes to the language. As a result, once we release a language feature, we are essentially restricted to backwards compatible changes. We've attempted to label a few features as "experimental" in the past, with a warning that they might change without notice: the result of this so far has been that very few people used these features, so we didn't get much meaningful feedback on the design from "real" users, which is a critical requirement for an agile process.

The idea that you can "educate" your way out of bad design in general and badly named features in particular doesn't seem like a good principle on which to base programming language design.