This week we will add a few more features to the bulk filing script. Now that Steve has filled in the code to create a bulk filing we can add the finishing touches to increase performance, check that the filing size is within acceptable limits, and improve the user interface. If you haven’t read the previous posts, here are the links to part one and two.
Friday, January 11. 2019
LDC #118: Bulk Filings Part 3
Let’s start with the user interface improvements. We received a request to add the contents of a folder to the list of filings. Each filing in the folder would be added to the bulk submission. In order to do this we need to add a button to the dialog as well as some code to the action function. Let’s look at the changes to the action function.
int bd_action(int id, int action) {
string tba[];
string path;
string ext;
int rc, ix, mx;
switch (id) {
...
case DC_ADDFOLDER:
path = BrowseFolder("Select Projects");
if (GetLastError() == ERROR_CANCEL) {
break;
}
tba = EnumerateFiles(path, FOLDER_LOAD_NO_FOLDER_NAV | FOLDER_LOAD_RECURSE);
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
ext = MakeLowerCase(GetExtension(tba[ix]));
if ((ext == ".gfp") || (ext == ".eis")) {
tba[ix] = AddPaths(path, tba[ix]);
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
}
bd_set_state();
break;
...
}
return ERROR_NONE;
}
The logic for this button is very similar to the add button. Instead of using BrowseOpenFiles we use BrowseFolder to allow the user to select the folder they wish to add. We then use the EnumerateFiles function to get an array of files within the folder and its children. Now that we have an array of files we can iterate over the list and add them to the list control. However, because we loaded all the files in the selected folder we need to filter the results to projects. We decided not to add XML files in the folders as too many EDGAR submissions have XML attachments. We do this by checking the extension of each file. We could change this to check the file type of each file but that requires opening each file to check the contents, which is a pretty slow operation. For now, we can rely on the user to select XML files individually.
We also improved the list control so when long names are added the contents of the list box scrolls. In order to accomplish this there is an additional line of code in the load function.
ControlSendMessage(DC_LIST, LB_SETHORIZONTALEXTENT, 800, 0);
Since Legato doesn’t have a function to set the horizontal extent of a list box we can use the ControlSendMessage to talk directly to the list box. In this case we are telling the list box that we want to be able to scroll to 800 pixels.
This covers all of the user interface improvements, so let’s now look at how logic for the bulk filing creation. First, to improve performance we have changed the logic to write the bulk filing directly to disk as it is created. This means that Legato isn’t keeping a very large file in memory as it’s being created. Previously, the file was stored in a string pool which is memory efficient but requires the whole bulk file to be in memory. Since we are writing the file in sequential parts, writing the contents to disk as we go reduces the memory requirements.
Second, starting with GoFiler 4.25a, Legato now offers EDGAR Access Object functions. These functions allow for a project to be opened, edited and saved all in the background. This means the user doesn’t see the application open a project. This increases performance as creating and displaying the user interface for the project editor is quite expensive.
Here is the updated run function:
int run(int id, string mode) {
int size;
int bytes;
int num_files;
int ix;
int rc;
int limit;
handle hResult;
handle log;
handle hAttachment;
handle hEDAC;
string fname;
string enc_fname;
string outfile;
string params;
string test;
string file;
string attachment;
if (mode != "preprocess") {
return ERROR_NONE;
}
// Load Settings
bulk = GetSetting("settings","bulk");
files = ExplodeString(GetSetting("settings","files"),"|");
// Run Dialog
rc = DialogBox("BulkDialog", "bd_");
if (IsError(rc)) {
return rc;
}
// Set up
log = LogCreate("Create Bulk Filing");
hResult = CreateFile(bulk);
if (IsError(hResult)) {
MessageBox('x', "Could not save to file. %s", TranslateWindowsError(GetLastError()));
return ERROR_FILE;
}
ProgressOpen("Creating Bulk Filing...", 0);
// Start of File
hAttachment = PoolCreate();
WriteLine(hResult, OPEN);
test = ReplaceInString(LIVE_FLAG, TEST_DIRECTIVE, live);
WriteLine(hResult, test);
// Write Attachments
limit = -1;
size = ArrayGetAxisDepth(files);
WriteBlock(hResult, ATTACHMENTS_OPEN, GetStringLength(ATTACHMENTS_OPEN));
for (ix = 0; ix < size; ix++) {
ProgressUpdate(ix, size);
ProgressSetStatus("File %d of %d", ix + 1, size);
PoolReset(hAttachment);
fname = files[ix];
outfile = AddPaths(GetTempFileFolder(), GetFilename(fname));
outfile = ClipFileExtension(outfile);
outfile = outfile + ".xml";
AddMessage(log, "Adding file %s to bulk filing", fname);
hEDAC = EDACOpenFile(fname);
if (IsError(hEDAC)) {
rc = GetLastError();
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot open filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
rc = EDACSaveFile(hEDAC, outfile);
CloseHandle(hEDAC);
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot save filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
file = FileToString(outfile);
rc = GetLastError();
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot add file to filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
// Build Attachment Code
file = EncodeString(file);
PoolAppend(hAttachment, ReplaceInString(ATTACHMENT_OPEN, NAME_DIRECTIVE, GetFilename(outfile)));
PoolAppend(hAttachment, file);
PoolAppend(hAttachment, ATTACHMENT_CLOSE);
// Write to File
attachment = PoolGetPool(hAttachment);
WriteBlock(hResult, attachment, PoolGetPosition(hAttachment));
DeleteFile(outfile);
bytes = GetFilePosition(hResult);
if (bytes >= SEC_FILE_LIMIT) {
limit = ix;
}
num_files++;
}
// Finish File
WriteLine(hResult, ATTACHMENTS_CLOSE);
WriteLine(hResult, CLOSE);
CloseHandle(hResult);
ProgressClose();
if (limit >= 0) {
LogSetMessageType(LOG_WARNING);
AddMessage(log, " Bulk filing size (%d) exceeds EDGAR transmission (%d) limit.", GetFileSize(hResult), SEC_FILE_LIMIT);
AddMessage(log, " Remove files after entry %d (%s).", limit + 1, files[limit]);
LogSetMessageType(LOG_NONE);
}
LogDisplay(log);
MessageBox('i', "Added %d projects to bulk file %s.", num_files, bulk);
return 0;
}
As you can see there are many changes, but the core logic stayed the same. I will only go over the changed parts of the code. First we need a few more variables to track which file was the one that caused the bulk submission to exceed the SEC file size limit. Additionally, we no longer have a pool variable but rather a result handle for the file. Finally, we also now have a handle for the EDGAR Access Object.
We now start by using CreateFile to create a file to save the bulk data. If this fails, we can no longer continue so we stop processing. Otherwise we create a progress window using the ProgressOpen function. Now that the main set up is out of the way we can prepare the start of the file. Instead of writing to the pool variable we now use WriteLine to write directly to the file. We also reset the limit variable for later.
In the loop we start off by updating the progress window before we reset the attachment pool. This is part of the performance improvements. In the previous version the attachment pool was for all attachments. In this version of the script the attachment pool is only for the code of a single attachment. Because of this it should be reset each time the loop happens. The outfile variable has been changed slightly to instead use the temp area. We don’t want our script to overwrite other XML files that may be in the folder of our project. Most people use separate folders for each project but if someone does not this could have been an issue.
Now the real changes begin, instead of using RunMenuFunction we use the EDACOpenFile function to open each project. As stated above, this operation is significantly faster. We then use the EDACSaveFile function to have the application save an XML version of the project to the temp area. Other than the error reporting this portion of the code is the same as the last blog. The only big difference is we write to the file using WriteBlock.
Now that the attachment has been written to the file we can look to see if we exceeded the SEC submission limit. We get the position we are currently writing in the file. Since we are writing sequentially this position is equivalent to the size of the file. If this size is too large we mark which file it is. This is important since instead of just telling the user that the bulk filing is too large we can tell the user which file caused the submission to exceed the limit. That way they don’t have to guess how many files they need to remove to get the file size to be correct. They can instead remove that file and all the files beyond it in the list to create a smaller submission.
This concludes the loop. We wrap up by writing the close information for the bulk submission as well as close the progress window. If the file size was too large we add an additional warning to the log stating as such using the limit variable.
As Steve stated in the last blog the script was a good starting point. Now it has been improved further. Now all we need is the logic to submit the filing. If you are looking for a fun challenge, see if you can add the logic!
Here is the entire script file:
//
// Function Definitions and Globals
// --------------------------------
#define TEST_DIRECTIVE "%%%TEST%%%"
#define NAME_DIRECTIVE "%%%NAME%%%"
#define OPEN "<?xml version=\"1.0\" ?>\r\n<bul:edgarBulkSubmission xmlns:bul=\"http://www.sec.gov/edgar/bulkfiling\">"
#define LIVE_FLAG "<bul:liveTestFlag>"+TEST_DIRECTIVE+"</bul:liveTestFlag>"
#define ATTACHMENTS_OPEN "<bul:attachments>"
#define ATTACHMENTS_CLOSE "\r\n</bul:attachments>"
#define ATTACHMENT_OPEN "\r\n<bul:attachment>\r\n<bul:submissionName>" + NAME_DIRECTIVE + "</bul:submissionName>\r\n<bul:contents>\r\n"
#define ATTACHMENT_CLOSE "</bul:contents>\r\n</bul:attachment>"
#define CLOSE "</bul:edgarBulkSubmission>"
#define SEC_FILE_LIMIT 209715200
int bd_load ();
void bd_set_state ();
int bd_action (int id, int action);
int bd_validate ();
string files[];
string bulk;
string live;
int run(int id, string mode);
void setup();
void main(){
setup();
}
void setup(){
string menu[];
menu["Code"] = "CREATE_BULK_FILING";
menu["MenuText"] = "Create Bulk Filing";
menu["Description"] = "Compresses multiple GFP files into a single BULK XML File.";
MenuAddFunction(menu);
MenuSetHook("CREATE_BULK_FILING",GetScriptFilename(),"run");
}
int run(int id, string mode) {
int size;
int bytes;
int num_files;
int ix;
int rc;
int limit;
handle hResult;
handle log;
handle hAttachment;
handle hEDAC;
string fname;
string enc_fname;
string outfile;
string params;
string test;
string file;
string attachment;
if (mode != "preprocess") {
return ERROR_NONE;
}
// Load Settings
bulk = GetSetting("settings","bulk");
files = ExplodeString(GetSetting("settings","files"),"|");
// Run Dialog
rc = DialogBox("BulkDialog", "bd_");
if (IsError(rc)) {
return rc;
}
// Set up
log = LogCreate("Create Bulk Filing");
hResult = CreateFile(bulk);
if (IsError(hResult)) {
MessageBox('x', "Could not save to file. %s", TranslateWindowsError(GetLastError()));
return ERROR_FILE;
}
ProgressOpen("Creating Bulk Filing...", 0);
// Start of File
hAttachment = PoolCreate();
WriteLine(hResult, OPEN);
test = ReplaceInString(LIVE_FLAG, TEST_DIRECTIVE, live);
WriteLine(hResult, test);
// Write Attachments
limit = -1;
size = ArrayGetAxisDepth(files);
WriteBlock(hResult, ATTACHMENTS_OPEN, GetStringLength(ATTACHMENTS_OPEN));
for (ix = 0; ix < size; ix++) {
ProgressUpdate(ix, size);
ProgressSetStatus("File %d of %d", ix + 1, size);
PoolReset(hAttachment);
fname = files[ix];
outfile = AddPaths(GetTempFileFolder(), GetFilename(fname));
outfile = ClipFileExtension(outfile);
outfile = outfile + ".xml";
AddMessage(log, "Adding file %s to bulk filing", fname);
hEDAC = EDACOpenFile(fname);
if (IsError(hEDAC)) {
rc = GetLastError();
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot open filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
rc = EDACSaveFile(hEDAC, outfile);
CloseHandle(hEDAC);
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot save filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
file = FileToString(outfile);
rc = GetLastError();
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot add file to filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
// Build Attachment Code
file = EncodeString(file);
PoolAppend(hAttachment, ReplaceInString(ATTACHMENT_OPEN, NAME_DIRECTIVE, GetFilename(outfile)));
PoolAppend(hAttachment, file);
PoolAppend(hAttachment, ATTACHMENT_CLOSE);
// Write to File
attachment = PoolGetPool(hAttachment);
WriteBlock(hResult, attachment, PoolGetPosition(hAttachment));
DeleteFile(outfile);
bytes = GetFilePosition(hResult);
if (bytes >= SEC_FILE_LIMIT) {
limit = ix;
}
num_files++;
}
// Finish File
WriteLine(hResult, ATTACHMENTS_CLOSE);
WriteLine(hResult, CLOSE);
CloseHandle(hResult);
ProgressClose();
if (limit >= 0) {
LogSetMessageType(LOG_WARNING);
AddMessage(log, " Bulk filing size (%d) exceeds EDGAR transmission (%d) limit.", GetFileSize(hResult), SEC_FILE_LIMIT);
AddMessage(log, " Remove files after entry %d (%s).", limit + 1, files[limit]);
LogSetMessageType(LOG_NONE);
}
LogDisplay(log);
MessageBox('i', "Added %d projects to bulk file %s.", num_files, bulk);
return 0;
}
//
// Dialog and Defines
// --------------------------------
#define DC_BULK 201
#define DC_LIST 301
#define DC_ADD 101
#define DC_ADDFOLDER 102
#define DC_REMOVE 103
#define DC_BROWSE 104
#define DC_LIVE 105
#define DC_CLEAR 106
#define ADD_FILTER "All Projects|*.gfp;*.eis;*.xml;&GoFiler Projects|*.gfp&EDGAR Internet Submissions|*.eis&XML Submission Files|*.xml&All Files *.*|*.*"
#define SAVE_FILTER "XML Submission Files|*.xml&All Files *.*|*.*"
int bd_load() {
int ix, mx;
// Load List
mx = ArrayGetAxisDepth(files);
for (ix = 0; ix < mx; ix++) {
ListBoxAddItem(DC_LIST, files[ix]);
}
ControlSendMessage(DC_LIST, LB_SETHORIZONTALEXTENT, 800, 0);
EditSetText(DC_BULK, bulk);
bd_set_state();
return ERROR_NONE;
}
void bd_set_state() {
int fx;
int ix;
fx = ListBoxGetSelectIndex(DC_LIST);
ix = ListBoxGetItemCount(DC_LIST);
if (ix > 0) {
ControlEnable(DC_CLEAR);
}
else {
ControlDisable(DC_CLEAR);
}
if (fx >= 0) {
ControlEnable(DC_REMOVE);
}
else {
ControlDisable(DC_REMOVE);
}
}
int bd_action(int id, int action) {
string tba[];
string path;
string ext;
int rc, ix, mx;
switch (id) {
case DC_ADD:
tba = BrowseOpenFiles("Select Projects", ADD_FILTER);
if (GetLastError() == ERROR_CANCEL) {
break;
}
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
bd_set_state();
break;
case DC_ADDFOLDER:
path = BrowseFolder("Select Projects");
if (GetLastError() == ERROR_CANCEL) {
break;
}
tba = EnumerateFiles(path, FOLDER_LOAD_NO_FOLDER_NAV | FOLDER_LOAD_RECURSE);
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
ext = MakeLowerCase(GetExtension(tba[ix]));
if ((ext == ".gfp") || (ext == ".eis")) {
tba[ix] = AddPaths(path, tba[ix]);
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
}
bd_set_state();
break;
case DC_REMOVE:
rc = ListBoxGetSelectIndex(DC_LIST);
if (rc >= 0) {
ListBoxDeleteItem(DC_LIST, rc);
while (rc >= ListBoxGetItemCount(DC_LIST)) {
rc--;
}
if (rc >= 0) {
ListBoxSetSelectIndex(DC_LIST, rc);
}
}
bd_set_state();
break;
case DC_CLEAR:
ListBoxReset(DC_LIST);
bd_set_state();
break;
case DC_BROWSE:
path = EditGetText(DC_BULK);
path = BrowseSaveFile("Save Bulk Submission", SAVE_FILTER, path);
if (GetLastError() == ERROR_CANCEL) {
break;
}
EditSetText(DC_BULK, path);
break;
case DC_LIST:
if (action == LBN_SELCHANGE) {
bd_set_state();
}
break;
}
return ERROR_NONE;
}
int bd_validate() {
string tba[];
string name;
string s1;
int ix;
int size;
// Check List
tba = ListBoxGetArray(DC_LIST);
if (ArrayGetAxisDepth(tba) == 0) {
MessageBox('x', "Submission must have at least one file.");
return ERROR_SOFT | DC_LIST;
}
// Check Name
name = EditGetText(DC_BULK);
if (name == "") {
MessageBox('x', "Bulk Submission File is a required field.");
return ERROR_SOFT | DC_LIST;
}
if (MakeLowerCase(GetExtension(name)) != ".xml") {
MessageBox('x', "Bulk Submission File must be an XML file");
return ERROR_SOFT | DC_LIST;
}
if (CanAccessFile(name) == false){
MessageBox('x', "Bulk Submission File must be a valid file location and not open by other applications.");
return ERROR_SOFT | DC_LIST;
}
// Save Values
bulk = name;
files = tba;
if (CheckboxGetState(DC_LIVE) == BST_CHECKED){
live = "LIVE";
}
else{
live = "TEST";
}
s1 = ImplodeArray(files, "|");
PutSetting("settings", "bulk", bulk);
PutSetting("settings", "files", s1);
return ERROR_NONE;
}
#beginresource
BulkDialog DIALOGEX 0, 0, 344, 190
EXSTYLE WS_EX_DLGMODALFRAME
STYLE DS_MODALFRAME | DS_3DLOOK | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "Bulk Filing"
FONT 8, "MS Shell Dlg", 400, 0
{
CONTROL "Files", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 4, 20, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 26, 9, 314, 1, 0
CONTROL "", DC_LIST, "listbox", LBS_NOTIFY | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_VSCROLL | WS_HSCROLL | WS_TABSTOP, 12, 16, 276, 120, 0
CONTROL "&Add...", DC_ADD, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 16, 45, 12, 0
CONTROL "Add &Folder...", DC_ADDFOLDER, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 31, 45, 12, 0
CONTROL "&Remove", DC_REMOVE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 46, 45, 12, 0
CONTROL "&Clear", DC_CLEAR, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 61, 45, 12, 0
CONTROL "&LIVE", DC_LIVE, "button", BS_AUTOCHECKBOX | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 296, 76, 45, 12, 0
CONTROL "Submission", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 137, 42, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 48, 142, 292, 1, 0
CONTROL "&Bulk Submission File:", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 12, 150, 72, 8, 0
CONTROL "", DC_BULK, "edit", ES_AUTOHSCROLL | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_TABSTOP, 88, 148, 200, 12
CONTROL "Browse...", DC_BROWSE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 148, 45, 12, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 6, 166, 335, 1, 0
CONTROL "&Create", IDOK, "BUTTON", BS_DEFPUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 232, 172, 50, 14
CONTROL "Cancel", IDCANCEL, "BUTTON", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 287, 172, 50, 14
}
#endresource
 David Theis has been developing software for Windows operating systems for over fifteen years. He has a Bachelor of Sciences in Computer Science from the Rochester Institute of Technology and co-founded Novaworks in 2006. He is the Vice President of Development and is one of the primary developers of GoFiler, a financial reporting software package designed to create and file EDGAR XML, HTML, and XBRL documents to the U.S. Securities and Exchange Commission. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
