When you file something to the SEC, there are a lot of data points that can be tracked from that event such as: if the filing was live, when the filing process began, when GoFiler began transmitting the file, and when GoFiler finished transmitting the file. This information can be gleaned different sources, including the log file and the project file metadata. However, there is no one place where filers can obtain this information. With Legato, we can set up a small script that triggers when we file a project, and simply records information about that project into a log file in the same folder as our project.
Friday, September 27. 2019
LDC #154: Enhanced Submission Logging
GoFiler 5.1a, released on September 27, 2019, adds some more functionality to this script. Our pre and post process hooks track when the user began the filing process and when the user ended the filing process, but that doesn’t really tell us too much about when the filing happened, other than that it was between those times. For example, if the user pressed the “FILE LIVE” button and walked away before confirming, well we don’t know exactly when the user pressed the “OK” button to confirm the action and GoFiler began transmitting. GoFiler 5.1a fixes that, and the GetMenuResponse function now returns the additional parameters “BeginTime” and “EndTime” that show when GoFiler actually began transmitting and ended transmitting. So if a user presses the button to begin a filing, doesn’t confirm it, then comes back later and hits the “OK” button, we can log that there was a delay between the process starting and when the filing started. While this information is only in the newest version of GoFiler, the script is still backwards compatible with older versions of the software, since we’re not explicitly retrieving parameters, but instead just enumerating them with array iterators.
Let’s look at our actual script. It’s quite simple, just two functions... setup and log. We’ll start with the setup function.
/****************************************/
void setup() { /* Called from Application Startup */
/****************************************/
string fn; /* file name */
string item[10]; /* menu item array */
/* */
fn = GetScriptFilename(); /* get the filename of the script */
MenuSetHook("EDGAR_SUBMIT_TEST", fn, "log"); /* add the hook */
MenuSetHook("EDGAR_SUBMIT_LIVE", fn, "log"); /* add the hook */
MenuSetHook("EDGAR_SUBMIT_TEST_AGENT", fn, "log"); /* add the hook */
} /* */
This is pretty similar to our other setup functions, but we need to make sure to hook it into all of our three primary submission methods. This function needs to trigger when a user runs a TEST, LIVE, or TEST as agent operation, so that it can gather information to put into our log file. Note that we’re not tracking NSS filings. NSS doesn’t have very much information in the menu function response, since the actual filing is not handled by this machine, but rather by the computer running NSS. If we want to cover NSS with our enhanced logging, we’d have to write a second script to log information, that would run on the NSS server. That’s a bit beyond the scope of this script though, so we’ll stick with just these three filing methods. Moving on, lets take a look at the log function.
/****************************************/
int log(int f_id, string mode){ /* Log the function response */
/****************************************/
... omitted variable declarations ...
if (mode == "preprocess"){ /* if we're in preprocess */
begin = GetUTCTime(DS_ISO_8601); /* get UTC time */
return ERROR_NONE; /* return no error */
} /* */
if (mode=="postprocess"){ /* only get postprocess */
end = GetUTCTime(DS_ISO_8601); /* get end time */
} /* */
s1 = GetMenuFunctionResponse(); /* get the response */
response = ParametersToArray(s1); /* parse response */
response["StartProcess"] = begin+"Z"; /* set process begin time */
response["EndProcess"] = end+"Z"; /* set end time */
param_names = ArrayGetKeys(response); /* get keys of response */
response["User"]= GetUserName(); /* get the username */
size = ArrayGetAxisDepth(response); /* get size of response */
Unlike most of our scripts, this one actually needs to do something in both preprocess and postprocess. The preprocess action is simple, but we need to take the timestamp of when the function actually begins. This should correspond with when the user begins the filing process. Once we have that, we can just return, because the real meat of the function happens postprocess. If we’re running in postprocess, we can record the end time, which should correspond with when the user presses the button to confirm that the filing was submitted, and then we can use the GetMenuFunctionResponse function to get the list of parameters this function returns. This is returned as a parameter string that GoFiler can parse, but an example of the returned values in an easier to read format would look like:
FormType: 10-Q
PrimaryCIK:
AgentCIK: 0000990681
LiveFiling: 0
File: C:\Users\steven.horowitz\Desktop\testbench.gfp
ValidatedOK: 1
AccessionNumber: 0000990681-19-000798
BeginTime: 2019-09-27T20:11:07Z
CompleteTime: 2019-09-27T20:11:10Z
InfoViewTab: 0x007002F0
InfoMessages: 9
TotalMessages: 9
Once we have our information from our function to log, we can convert the parameter string into an array with ParametersToArray to make it easier to work with, and then add our recorded start and end times, as well as the current user’s name, to our response array. After the response array is all set, we can use ArrayGetKeys to get the names of each of these fields, so we’re left with a pair of arrays that have the same number of elements, one has the key names for each data point, and the other has the value of each data point.
filename = response["File"]; /* get the filename of the project */
filename = ClipFileExtension(filename); /* clip the extension off */
filename = filename + ".log"; /* add log extension */
/* */
if(IsFile(filename)){ /* if the file exists */
mtext = OpenMappedTextFile(filename); /* open file as mapped text */
InsertLine(mtext,-1,""); /* insert blank line */
submissions = ReadLine(mtext,0); /* get first line of text */
submissions = GetParameter(submissions,"Submissions"); /* get number of submissions */
num_subs = TextToInteger(submissions); /* get the number of submissions */
num_subs++; /* increment number of submissions */
ReplaceLine(mtext,0,FormatString("Submissions:%d;",num_subs)); /* change number of submissions */
} /* */
Now that we have our data, we need to get the name of our log file by modifying the name of the project file we submitted. If there is already a log file by that name, we can open it as a Mapped Text Object to make working with it a lot easier. The first thing we’d need to do to our Mapped Text Object is to add a new line as a spacer with InsertLine for readability. Next, we can read the first line of the file, and get the number of submissions this file has logged. We can then increment it, because we’ve obviously just done one more, and then set that value back into our Mapped Text Object so when we write the file back out it will have the new updated number of submissions.
else{ /* if mapped text doesn't exist */
mtext = CreateMappedTextFile(filename); /* create new mapped text file */
submissions = "Submissions:1;"; /* set submissions string */
ReplaceLine(mtext,0,submissions); /* insert submissions string */
InsertLine(mtext,-1,""); /* insert blank line into mtext */
num_subs = 1; /* set number of submissions */
}
pre = FormatString("Submission %d ",num_subs); /* set prefix for submission line */
for(ix=0; ix<size; ix++){ /* for each item in response */
InsertLine(mtext,-1, /* insert into mtext */
FormatString("%s%s: %s",pre,param_names[ix],response[ix])); /* format string for mtext */
} /* */
MappedTextSave(mtext); /* save mapped text object */
return ERROR_NONE; /* return */
} /* */
If we don’t have a log file already created, we can create one with CreateMappedTextFile, and add some primer information to it, like our submissions count line and a spacer line. In either case, we then need to create a prefix for our log information to differentiate submissions, and then for each item in our response array, use InsertLine to insert it into the Mapped Text Object. After we’ve finished processing the data, we can call MappedTextSave and return, because our log file is now updated.
This is an example of what our finished log file would look like:
Submissions:3; Submission 1 FormType: 10-Q Submission 1 PrimaryCIK: Submission 1 AgentCIK: 0000990681 Submission 1 LiveFiling: 0 Submission 1 File: C:\Users\steven.horowitz\Desktop\testbench.gfp Submission 1 ValidatedOK: 1 Submission 1 AccessionNumber: 0000990681-19-000798 Submission 1 BeginTime: 2019-09-27T20:11:07Z Submission 1 CompleteTime: 2019-09-27T20:11:10Z Submission 1 InfoViewTab: 0x007002F0 Submission 1 InfoMessages: 9 Submission 1 TotalMessages: 9 Submission 1 StartProcess: 2019-09-27T20:11:04Z Submission 1 EndProcess: 2019-09-27T20:11:28Z Submission 1 User: steven.horowitz Submission 2 FormType: 10-Q Submission 2 PrimaryCIK: Submission 2 AgentCIK: 0000990681 Submission 2 LiveFiling: 0 Submission 2 File: C:\Users\steven.horowitz\Desktop\testbench.gfp Submission 2 ValidatedOK: 1 Submission 2 AccessionNumber: 0000990681-19-000799 Submission 2 BeginTime: 2019-09-27T20:12:57Z Submission 2 CompleteTime: 2019-09-27T20:13:00Z Submission 2 InfoViewTab: 0x000D1EA6 Submission 2 InfoMessages: 9 Submission 2 TotalMessages: 9 Submission 2 StartProcess: 2019-09-27T20:12:54Z Submission 2 EndProcess: 2019-09-27T20:13:02Z Submission 2 User: steven.horowitz Submission 3 FormType: 10-Q Submission 3 PrimaryCIK: Submission 3 AgentCIK: 0000990681 Submission 3 LiveFiling: 0 Submission 3 File: C:\Users\steven.horowitz\Desktop\testbench.gfp Submission 3 ValidatedOK: 1 Submission 3 AccessionNumber: 0000990681-19-000800 Submission 3 BeginTime: 2019-09-27T20:13:15Z Submission 3 CompleteTime: 2019-09-27T20:13:19Z Submission 3 InfoViewTab: 0x00FD02DC Submission 3 InfoMessages: 9 Submission 3 TotalMessages: 9 Submission 3 StartProcess: 2019-09-27T20:13:11Z Submission 3 EndProcess: 2019-09-27T20:14:19Z Submission 3 User: steven.horowitz
Here’s our completed script, without any commentary:
int log(int f_id, string mode); /* */
/* */
string begin; /* beginning file time */
string end; /* ending file time */
/* */
/****************************************/
void setup() { /* Called from Application Startup */
/****************************************/
string fn; /* file name */
string item[10]; /* menu item array */
/* */
fn = GetScriptFilename(); /* get the filename of the script */
MenuSetHook("EDGAR_SUBMIT_TEST", fn, "log"); /* add the hook */
MenuSetHook("EDGAR_SUBMIT_LIVE", fn, "log"); /* add the hook */
MenuSetHook("EDGAR_SUBMIT_TEST_AGENT", fn, "log"); /* add the hook */
} /* */
/****************************************/
int log(int f_id, string mode){ /* Log the function response */
/****************************************/
handle mtext; /* handle to mapped text object */
string submissions; /* submissions header string */
int num_subs; /* number of submissions */
string output; /* output string */
string response[]; /* sorted response */
string param_names[]; /* names of paramters */
string pre; /* prefix for submission lines */
string filename; /* filename of log */
int size,ix; /* counter variables */
string row; /* a row of the response */
string s1; /* response */
/* */
if (mode == "preprocess"){ /* if we're in preprocess */
begin = GetUTCTime(DS_ISO_8601); /* get UTC time */
return ERROR_NONE; /* return no error */
} /* */
if (mode=="postprocess"){ /* only get postprocess */
end = GetUTCTime(DS_ISO_8601); /* get end time */
} /* */
s1 = GetMenuFunctionResponse(); /* get the response */
response = ParametersToArray(s1); /* parse response */
response["StartProcess"] = begin+"Z"; /* set process begin time */
response["EndProcess"] = end+"Z"; /* set end time */
response["User"]= GetUserName(); /* get the username */
param_names = ArrayGetKeys(response); /* get keys of response */
size = ArrayGetAxisDepth(response); /* get size of response */
filename = response["File"]; /* get the filename of the project */
filename = ClipFileExtension(filename); /* clip the extension off */
filename = filename + ".log"; /* add log extension */
/* */
if(IsFile(filename)){ /* if the file exists */
mtext = OpenMappedTextFile(filename); /* open file as mapped text */
InsertLine(mtext,-1,""); /* insert blank line */
submissions = ReadLine(mtext,0); /* get first line of text */
submissions = GetParameter(submissions,"Submissions"); /* get number of submissions */
num_subs = TextToInteger(submissions); /* get the number of submissions */
num_subs++; /* increment number of submissions */
ReplaceLine(mtext,0,FormatString("Submissions:%d;",num_subs)); /* change number of submissions */
} /* */
else{ /* if mapped text doesn't exist */
mtext = CreateMappedTextFile(filename); /* create new mapped text file */
submissions = "Submissions:1;"; /* set submissions string */
ReplaceLine(mtext,0,submissions); /* insert submissions string */
InsertLine(mtext,-1,""); /* insert blank line into mtext */
num_subs = 1; /* set number of submissions */
}
pre = FormatString("Submission %d ",num_subs); /* set prefix for submission line */
for(ix=0; ix<size; ix++){ /* for each item in response */
InsertLine(mtext,-1, /* insert into mtext */
FormatString("%s%s: %s",pre,param_names[ix],response[ix])); /* format string for mtext */
} /* */
MappedTextSave(mtext); /* save mapped text object */
return ERROR_NONE; /* return */
} /* */
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
