Bulk extraction of all workspaces used in a legacy app

[viguice]

Hello,

I have to manage legacy applications written in APL for my company. The app is composed by more than 100 separate workspaces (mainly one for each input source, one that combine all the cleaned data and few other ones used to disseminate and build GUIs for end-users).
As I am a beginner in APL, I don't want to change the core engine for now but I would like at least to push everything in a git. I noticed that years after year, some element have been stored/duplicated in different places. So, I would like to do some clean-up but with the possibility to track and restore if something goes wrong. In this context, pushing everything into a git would be a big plus.
As I have a lot of workspaces involved, I cannot do by hand. So, I tried to due first with dyalogscript but I didn't succeed. So I have created a batch file as follows:

Code: Select all

#!/bin/bash
INDIR="$PWD/_SANDBOX_"
OUTDIR="$PWD/_SANDBOX_UNICODE_"
rm -r $OUTDIR
for ws in `find $INDIR -type f \( -name "*.dws" -o -name "*.DWS" \)`
do
  echo "WS: $ws"
  NEWDIR=$(dirname -- "$ws")
  NEWDIR=${NEWDIR/$INDIR/$OUTDIR}
  echo $NEWDIR
  mkdir -p $NEWDIR
  cd $NEWDIR
  filename=$(basename -- "$ws")
  extension="${filename##*.}"
  WS_name="${filename%.*}"

  echo $WS_name
  /usr/bin/dyalog -tty << END_OF_FILE
  )xload $ws
  ]SNAP $NEWDIR/ -makedir
  ⍝ OK but no file generated to reload the ws
  ⍝ ]SNAP $NEWDIR/ -makedir –loadfn
  ⍝* Command Execution Failed: Unable to create file /home/viguice/Documents/_SAND
  ⍝    BOX_UNICODE_/ -makedir –loadfn/=AMECOINIT=.dyalog: /home/viguice/Document
  ⍝    s/_SANDBOX_UNICODE_/ -makedir –loadfn/=AMECOINIT=.dyalog: Unable to creat
  ⍝    e file
  ⍝ Based on https://dyalog.tv/Dyalog18/?v=w4Wp01-d3Rw but not working even using batch mode
  ⍝]CreateProject $NEWDIR/$WS_name #
  )OFF
END_OF_FILE

done

The unix script loops in a source folder and do a specific action for each workspace. It is working but –loadfn option fails and I didn't succeed to generate a load_ws.dyalog file with this command. Do you know how to fix that?
]SNAP is still the best method for extracting all info from a workspace? I tried CreateProject but it seems to be interactive and I need to do in bulk (the batch mode was not working in my case).

[Vince|Dyalog]

Hi viguice,

I shall try this out and write an email to you from Dyalog Support.

Regards,

Vince

[paulmansour]

Hi,

Looks like an excellent plan to get the legacy app under source control. I assume you want to put it all into one workspace managed by one git project (that is what I would do).

Why not just copy all the workspaces into one workspace, and then use Link (or something link Dado) to write out the workspace as text files in a single folder? (Do not use ]SNAP, it is obsolete).

First, something like:

Code: Select all

CopyInAllWorkspaces←{
     ⍝ ⍵ ←→ Folder
     f←(⎕NINFO⍠1)⍵,'\*.dws' ⍝ All workspaces
     n←1⊃¨⎕NPARTS↑f          ⍝ Short file names
     s←⍎¨n ⎕NS¨⊂''          ⍝ Create namespaces
     _←s.⎕CY¨f               ⍝ Copy in code
     0
 }

Then, just use Link to write out the folder.

I would also not put all the namespaces in the root. I would create a top level namespace for your application, and then put the 100 namespaces under that. If you start cleaning up and refactoring in a big way, or even enhancing the legacy app, you are likely to extract some code as a dependency, or use some other packages, and that will make it all neater and easier to manage. To do this, simply make new a namespace and copy the function above into the new namespace and run it from there.

Note ⎕IO is 0 and ⎕ML is 1 for above code.
Note also that the short file names must be valid namespace names.

Sounds like it could be a fun project. Good luck!

[AndyS|Dyalog]

Which version of Dyalog are you using ?

I am assuming that you're running on Linux ..

You say that you're dealing with "legacy applications written in APL" .. depending on how old some of your workspaces are, you may be better off not looking for .dws or .DWS files, but rather use something like

Code: Select all

find . -exec file {} \; | grep -i "dyalog.*workspace" | sed 's/:.*$//'

as until 15.0 workspaces on non-Windows platforms were not saved with any file extension, and even now in the latest version (19.0) you can set WSEXT to anything you like. It's safer therefore to look at the magic number of each file.

In the unlikely event that you're running on AIX, you'll need to run something like

Code: Select all

find .  -exec file -m /opt/mdyalog/19.0/64/unicode/p9/magic {} \; |\
   grep -i "dyalog.*workspace" | sed 's/:.*$//'

as on AIX you have to use the copy of magic in the Dyalog installation directory, rather than being able to rely on the operating-system supplied one.

[viguice]

Thanks for your feedback to both of you.

Paul:

My first trial was based on your presentation of ]CreateProject during Dyalog'18 user meeting but it doesn't react as you did. Now, it is asking me confirmation for the creation and opens the cider_config file. Not suitable in my case for running in batch mode and I gave up with this option. Can you tell if ]Link is a better option than ]CreateProject?
At least, it is sure that I have better results using ]Link that I had with ]SNAP.
So, I have adapted my code as follows:

Code: Select all

#!/bin/bash
INDIR="$PWD/_SANDBOX_"
OUTDIR="$PWD/_SANDBOX_UNICODE_"
LOGFILE="$PWD/link_all.log"
TEMPFILE="$PWD/link_temp.log"
ERRFILE="$PWD/link_err.log"

rm -r $OUTDIR
rm $LOGFILE $ERRFILE
OIFS="$IFS"
IFS=$'\n'
for ws in `find $INDIR -type f \( -name "*.dws" -o -name "*.DWS" \)`
do
  echo "WS: $ws"
  echo "WS: $ws" >> $LOGFILE
  NEWDIR=$(dirname -- "$ws")
  NEWDIR=${NEWDIR/$INDIR/$OUTDIR}
  echo $NEWDIR
  mkdir $NEWDIR
  cd $NEWDIR
  filename=$(basename -- "$ws")
  extension="${filename##*.}"
  WS_name="${filename%.*}"
  /usr/bin/dyalog -tty << END_OF_FILE > $TEMPFILE
  ⎕PW←256
  )xload "$ws"
  ]Link.export # "./WS_$WS_name" -arrays
  )OFF
END_OF_FILE
  if grep -q "File name case clash"  "$TEMPFILE"; then
    echo "-> Force casecode" >> $LOGFILE
    echo "   Issue for $(grep "^#." $TEMPFILE | tr -d ' ' | tr '\n' ' ')" >> $LOGFILE
    /usr/bin/dyalog -tty << END_OF_FILE2 > $TEMPFILE
    ⎕PW←256
    )xload "$ws"
    ]Link.export # "./WS_$WS_name" -arrays -casecode
    )OFF
END_OF_FILE2
  fi
  if grep -q "ERROR"  "$TEMPFILE"; then
    echo "WS: $ws" >> $ERRFILE
    cat $TEMPFILE >> $ERRFILE
    echo "-> Error for $(grep "^#." $TEMPFILE | tr '\n' ' ')" >> $LOGFILE
    echo "   Retry without these objects..." >> $LOGFILE
    rm -r ./WS_$WS_name
    /usr/bin/dyalog -tty << END_OF_FILE3 > $TEMPFILE
    ⎕PW←256
    )xload "$ws"
    )erase $(grep "^#." $TEMPFILE | tr '\n' ' ')
    ]Link.export # "./WS_$WS_name" -arrays -casecode
    )OFF
END_OF_FILE3
    if grep -q "ERROR"  "$TEMPFILE"; then
      echo "WS: $ws" >> $ERRFILE
      cat $TEMPFILE >> $ERRFILE
      echo "   Still not working..." >> $LOGFILE     
    fi
  else
    cat $TEMPFILE >> $LOGFILE
  fi
  /usr/bin/dyalog -tty << END_OF_FILE4
  ⍝ ⎕SE.Link.Import # ''WS_$WS_name'' -overwrite not working
  ⍝ #.⎕FX'∆WS_LX' '⎕SE.Link.Import # ''WS_$WS_name'''
  ⍝ #.⎕FX'∆WS_LX' '⎕SE.Link.Import # (⎕WSID,WS_$WS_name)'
  #.⎕FX'∆WS_LX' '⎕SE.Link.Import # ''$NEWDIR/WS_$WS_name'''
  )SAVE $WS_name
  )OFF
END_OF_FILE4
done
IFS="$OIFS"
rm $TEMPFILE
grep -v "Opening in existing browser session." $LOGFILE
grep -v "Opening in existing browser session."  $ERRFILE

The shell script looks for all workspaces in the tree in input folder and creates a new WS_<migated> folder at the same location in the output folder with all the content saved at separate files.
Now, I tried to generate a workspace file linked to the folder but it works only on the command line. It seems that there is currently an issue with Create (https://github.com/Dyalog/link/issues/654). So I tried to use import instead. For backward compatibility with the current setup, the idea was to create an "empty" workspace with one unique function ∆WS_LX used when I open the workspace.
I have two issues with this:
- I would like to use relative paths. It works if I do interactively:

      ]CD 'my/path'
⎕SE.Link.Import # 'WS_<myworkspace>'

But I tried to concatenate for linking the ressource folder based on location of the current workspace and it doesn't seem to work. The expression

      ⎕SE.Link.Import #(⎕WSID,'/<myworkspace>')

says that the path is not correct even it looks ok.
If I hardcode the path, it works. I can open the workspace and everything is reimported. This is mainly due to my inexperience in APL but I didn't manage to have something dynamic.
- My second issue is that the overwrite option seems not to work properly. As soon as save my workspace, it fails afterwards to open because the objects are already in the workspace. We can supposedly overcome this issue using overwrite option but it seems not be recognized as a valid option.

On the general strategy, this is indeed my goal to merge them into one workspace with separated namespaces if possible. But as they are interlinked somehow, I need to process step by step. Merging them from will break all the dependencies.

Andy:
The original project in production is running on Dyalog 17 Classic on Windows. I have created a sandbox on Ubuntu with Dyalog 19 to try to extract the content. I don't know what exactly you mean by not looking to wfs files but my files are wfs files. There is no other type of file and your command gives me the same output. But it helps me for giving you more details on the original files: They are all 32-bit classic little-endian version 17.8. I am attending to recreate them into 64-bit unicode little-endian version 19.9 with code managed in a git.

[paulmansour]

Regarding ]CreateProject, it looks like that is a Cider command you are using, which is not what was demonstrated back in 2018, which was Acre. Cider is a project management application that uses Link. If it sounds confusing, that is because it is. I can't really offer any assistance with Cider or Link as I don't use them (I use Dado, a successor of Acre).

My guess is you might want to use Cider, and then not worry about Link.

I don't really understand the shell script, or why you using a shell script vs a simple APL function, or why you are saving a workspace at all, (though you might have good reason for all of that) so I'm not sure I can offer help more than the original function I wrote and the advice to use something like Link or Cider or Dado to write out the code as text files and dispense with the workspaces. But I'll try to answer any further questions I can.

[Vince|Dyalog]

Hi viguice,
a) You can read more about Cider here, https://github.com/aplteam/Cider

b) You can trace into ⎕SE.Link.Import to see what is happening when you do
⎕SE.Link.Import # 'WS_<myworkspace>'
compared with:
⎕SE.Link.Import #(⎕WSID,'/<myworkspace>')

What is in ⎕WSID when you are doing it non-interactively?

For a user coming from MS Windows, tracing this will be easier if you use RIDE. https://github.com/Dyalog/ride

c) Here is an illustration of how to set the overwrite option for ⎕SE.Link.Import:

      ⎕SE.Link.Import #'D:\temp\linktest'
⎕SE.Link.Import: Names already exist: use -overwrite flag to force overwriting the following names:
#.myns
#.ns
#.ns.foo
#.ns.subns
#.ns.subns.foo
#.opand
#.foo
      ⎕SE.Link.Import #'D:\temp\linktest'
      ∧
      options←⎕NS ⍬                                     ⍝ create empty namespace
      options.overwrite←1 
      options ⎕SE.Link.Import #'D:\temp\linktest'
Imported: # ← D:\temp\linktest

Regards,

Vince