This blog is the first part of a two part series. This first part will discuss how to change the EDGAR Preferences in Legato and show an example script to change the Filing Agent credentials automatically when a project is opened. The second part will show how to secure the script to prevent users from accessing any private information needed by the script. Be advised that the script in this first part is usable as-is, but the second part adds crucial security to prevent users from reading the agent credentials.
Friday, March 17. 2017
LDC #26: Automatically Changing the EDGAR Preferences Part 1
This blog is the first part of a two part series. This first part will discuss how to change the EDGAR Preferences in Legato and show an example script to change the Filing Agent credentials automatically when a project is opened. The second part will show how to secure the script to prevent users from accessing any private information needed by the script. Be advised that the script in this first part is usable as-is, but the second part adds crucial security to prevent users from reading the agent credentials.
We’re going to be building on quite a few previous topics this week. Our sample script uses multiple menu hooks. Menu hooks have been discussed in a previous blog post. The script also uses a small amount of regular expressions. For more information on regular expressions refer to this blog post.
// Predefines int setup (); string read_cik (); string read_cik_offset (handle edit_window, string name, int offy, int offx); string read_cik_formd (handle edit_window); string read_cik_mfp (handle edit_window); string get_agent_table (); int load_agent_table (); int on_open_project (int f_id, string mode); int on_new_project (int f_id, string mode); // Set up int setup() { string fnScript; // Get Script fnScript = GetScriptFilename(); // Hook Project Open MenuSetHook(MenuFindFunctionID("PSEUDO_OPENED_PROJECT"), fnScript, "on_open_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_PROJECT"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_SUBMISSION"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_13F"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_13H"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_17A"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_17H"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_C"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_D"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_MA"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_N-MFP"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_SDR"), fnScript, "on_new_project"); return ERROR_NONE; } // Read CIK from Open Project string read_cik() { dword window_type; handle edit_window; edit_window = GetEditWindowHandle(); if (IsError(edit_window)) { return ""; } window_type = GetEditWindowType(edit_window); window_type &= EDX_TYPE_ID_MASK; if (window_type == EDX_TYPE_EDGAR_VIEW) { return read_cik_offset(edit_window, "section_filer", 3, 1); } if (window_type == EDX_TYPE_XML_13F_VIEW) { return read_cik_offset(edit_window, "cik", 0, 0); } if (window_type == EDX_TYPE_XML_13H_VIEW) { return read_cik_offset(edit_window, "filerId", 0, 0); } // EDX_TYPE_XML_17_VIEW - Not defined in current SDK version if (window_type == 0x000001E0) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_17H_VIEW - Not defined in current SDK version if (window_type == 0x00000210) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_C_VIEW - Not defined in current SDK version if (window_type == 0x000001D0) { return read_cik_offset(edit_window, "cik", 0, 0); } if (window_type == EDX_TYPE_XML_D_VIEW) { return read_cik_formd(edit_window); } if (window_type == EDX_TYPE_XML_MA_VIEW) { return read_cik_offset(edit_window, "filerId", 0, 0); } if (window_type == EDX_TYPE_XML_MFP_VIEW) { return read_cik_mfp(edit_window); } // EDX_TYPE_XML_SDR_VIEW - Not defined in current SDK version if (window_type == 0x000001A8) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_RGA_VIEW - Not defined in current SDK version if (window_type == 0x000001C0) { return read_cik_offset(edit_window, "cik", 0, 0); } return ""; } // Read CIK using Offset string read_cik_offset(handle edit_window, string name, int offy, int offx) { string field; handle dataview; int location[2]; dataview = DataViewGetObject(edit_window, 0); field = DataViewFindCellByName(dataview, name); if (field != "") { location = CellAddressToIndex(field); return DataViewCellGetText(dataview, location[0] + offy, location[1] + offx); } return ""; } // Special read CIK from Form D string read_cik_formd(handle edit_window) { string field; handle dataview; int location[2]; dataview = DataViewGetObject(edit_window, 1); field = DataViewFindCellByName(dataview, "primaryIssuer"); if (field != "") { location = CellAddressToIndex(field); return DataViewCellGetText(dataview, location[0] + 2, location[1]); } return ""; } // Special read CIK from Form N-MFP string read_cik_mfp(handle edit_window) { string field; handle dataview; int location[2]; dataview = DataViewGetObject(edit_window, 0); field = DataViewFindCellByName(dataview, "filer:cik"); if (field != "") { location = CellAddressToIndex(field); field = DataViewCellGetText(dataview, location[0], location[1]); if (field != "") { return field; } } dataview = DataViewGetObject(edit_window, 4); field = DataViewFindCellByName(dataview, "filer:cik"); if (field != "") { location = CellAddressToIndex(field); field = DataViewCellGetText(dataview, location[0], location[1]); if (field != "") { return field; } } return ""; } // Globals for Table string ciks[]; string cccs[]; string passwords[]; int default_cik; // Get Agent Table string get_agent_table() { return FileToString(AddPaths(GetScriptFolder(), "agents.csv")); } int load_agent_table() { string datafile; string lines[]; string values[]; string s1; int size, ix; // Load Agent Preferences List datafile = get_agent_table(); if (datafile == "") { MessageBox('x', "Could not read data file 0x%08X", GetLastError()); return ERROR_EOD; } default_cik = 0; lines = ExplodeString(datafile); size = ArrayGetAxisDepth(lines); for (ix = 0; ix < size; ix++) { values = ExplodeString(lines[ix], ","); if (values[0] == "") { break; } s1 = ReplaceInStringRegex(values[0], "(\\d+)$", "0000000000$1"); ciks[ix] = ReplaceInStringRegex(s1, "0*(\\d{10})$", "$1"); cccs[ix] = values[1]; passwords[ix] = values[2]; if (values[3] != "") { default_cik = ix; } } return ERROR_NONE; } // Hook for Project Open int on_open_project(int f_id, string mode) { string s1; int size, rc, ix; if (mode != "postprocess") { return ERROR_NONE; } rc = load_agent_table(); if (IsError(rc)) { return rc; } // Get Filer CIK s1 = read_cik(); size = ArrayGetAxisDepth(ciks); ix = 0; while (ix < size) { if (s1 == ciks[ix]) { break; } ix++; } if (ix >= size) { ix = default_cik; } if (ix != default_cik) { rc = YesNoBox("Change Agent CIK to match Filer CIK?"); if (rc != IDYES) { ix = default_cik; } } // Set them rc = EDGARSetCredentials(ciks[ix], passwords[ix], cccs[ix]); if (IsError(rc)) { MessageBox('x', GetLastErrorMessage()); } return ERROR_NONE; } // Hook for New Project int on_new_project(int f_id, string mode) { dword window_type; handle edit_window; string s1; int size, rc, ix; if (mode != "postprocess") { return ERROR_NONE; } // Check Type edit_window = GetEditWindowHandle(); if (IsError(edit_window)) { return ERROR_NONE; } window_type = GetEditWindowType(edit_window); window_type &= EDX_TYPE_ID_MASK; if ((window_type != EDX_TYPE_EDGAR_VIEW) && (window_type != EDX_TYPE_XML_13F_VIEW) && (window_type != EDX_TYPE_XML_13H_VIEW) && // EDX_TYPE_XML_17_VIEW - Not defined in current SDK version (window_type != 0x000001E0) && // EDX_TYPE_XML_17H_VIEW - Not defined in current SDK version (window_type != 0x00000210) && // EDX_TYPE_XML_C_VIEW - Not defined in current SDK version (window_type != 0x000001D0) && (window_type != EDX_TYPE_XML_D_VIEW) && (window_type != EDX_TYPE_XML_MA_VIEW) && (window_type != EDX_TYPE_XML_MFP_VIEW) && // EDX_TYPE_XML_SDR_VIEW - Not defined in current SDK version (window_type != 0x000001A8) && // EDX_TYPE_XML_RGA_VIEW - Not defined in current SDK version (window_type != 0x000001C0)) { return ERROR_NONE; } rc = load_agent_table(); if (IsError(rc)) { return rc; } // Set to Default rc = EDGARSetCredentials(ciks[default_cik], passwords[default_cik], cccs[default_cik]); if (IsError(rc)) { MessageBox('x', GetLastErrorMessage()); } return ERROR_NONE; } int main() { setup(); return ERROR_NONE; }
This script entails many functions, some of which are programmer-defined. We have used programmer-defined functions in previous blog posts but without much explanation. Programmer-defined functions in Legato are similar to other programming languages. They do not necessarily require prototypes at the top of the script file or before they are used if they are ordered properly in the script itself. However, in the interests of good programming practices and clarity, we begin our script this week with our function prototypes.
// Predefines int setup (); string read_cik (); string read_cik_offset (handle edit_window, string name, int offy, int offx); string read_cik_formd (handle edit_window); string read_cik_mfp (handle edit_window); string get_agent_table (); int load_agent_table (); int on_open_project (int f_id, string mode); int on_new_project (int f_id, string mode);
We also have our usual main and setup functions. Our setup function in this case is going to get our script name and hook some of our programmer-defined functions into particular menu items using the MenuSetHook and the MenuFindFunctionID SDK functions. As we have covered in previous blog posts, you must use a unique ID to indicate a specific menu function. Last week’s article detailed a script that enumerated GoFiler’s menu functions, their descriptions, and their IDs.
// Get Script fnScript = GetScriptFilename(); // Hook Project Open MenuSetHook(MenuFindFunctionID("PSEUDO_OPENED_PROJECT"), fnScript, "on_open_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_PROJECT"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_SUBMISSION"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_13F"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_13H"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_17A"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_17H"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_C"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_D"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_MA"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_N-MFP"), fnScript, "on_new_project"); MenuSetHook(MenuFindFunctionID("FILE_NEW_XML_FORM_SDR"), fnScript, "on_new_project");
Now let’s look at our functions we are hooking into the menu functions. These are our main workhorses for this script. We have two: on_new_project and on_open_project. Both these functions perform similar tasks: loading EDGAR settings from a file, trying to match a project’s CIK to those parameters, and then setting the pertinent information in the EDGAR preferences in the application. Let’s examine the on_open_project function first.
// Hook for Project Open int on_open_project(int f_id, string mode) { string s1; int size, rc, ix; if (mode != "postprocess") { return ERROR_NONE; } rc = load_agent_table(); if (IsError(rc)) { return rc; } // Get Filer CIK s1 = read_cik(); size = ArrayGetAxisDepth(ciks); ix = 0; while (ix < size) { if (s1 == ciks[ix]) { break; } ix++; } if (ix >= size) { ix = default_cik; } if (ix != default_cik) { rc = YesNoBox("Change Agent CIK to match Filer CIK?"); if (rc != IDYES) { ix = default_cik; } } // Set them rc = EDGARSetCredentials(ciks[ix], passwords[ix], cccs[ix]); if (IsError(rc)) { MessageBox('x', GetLastErrorMessage()); } return ERROR_NONE; }
Our function takes the function ID as a int and the mode as a string for arguments. Immediately we check the mode and return if the script is not being called in the post process stage. See this previous blog post for more discussion about the different stages during which a hooked script may be called when the user clicks a menu function. After we clear this check, we call another programmer-defined function, load_agent_table. Let’s take a look inside this function.
// Globals for Table string ciks[]; string cccs[]; string passwords[]; int default_cik; int load_agent_table() { string datafile; string lines[]; string values[]; string s1; int size, ix; // Load Agent Preferences List datafile = get_agent_table(); if (datafile == "") { MessageBox('x', "Could not read data file 0x%08X", GetLastError()); return ERROR_EOD; } default_cik = 0; lines = ExplodeString(datafile); size = ArrayGetAxisDepth(lines); for (ix = 0; ix < size; ix++) { values = ExplodeString(lines[ix], ","); if (values[0] == "") { break; } s1 = ReplaceInStringRegex(values[0], "(\\d+)$", "0000000000$1"); ciks[ix] = ReplaceInStringRegex(s1, "0*(\\d{10})$", "$1"); cccs[ix] = values[1]; passwords[ix] = values[2]; if (values[3] != "") { default_cik = ix; } } return ERROR_NONE; }
Note the globally defined string arrays ciks, cccs, and passwords. These are where we will store the filer information we are reading from our CSV file. Because they are globally defined (not defined within any particular function), these arrays will be accessible to other functions, which is what we need. Our load_agent_table function also returns an integer error code to its caller. Since this function performs the key act of reading and parsing our filer data, we need to know if it was successful before proceeding.
First, this function calls another function we have written, get_agent_table, which simply returns a string version of our CSV file containing the filer information. Since this function is a single line the code could be written in place of the function call. But we made it a function now so we could expand it in the next blog post.
// Get Agent Table string get_agent_table() { return FileToString(AddPaths(GetScriptFolder(), "agents.csv")); }
We’ve hard-coded the location (the script folder) and the name of our CSV file, though there are more elaborate ways to inquire from the user about which file to use. The load_agent_table function checks to see if the get_agent_table function returns an empty string, and, if so, stops execution and reports the error. If not, we start parsing our CSV string.
default_cik = 0; lines = ExplodeString(datafile); size = ArrayGetAxisDepth(lines); for (ix = 0; ix < size; ix++) { values = ExplodeString(lines[ix], ","); if (values[0] == "") { break; } s1 = ReplaceInStringRegex(values[0], "(\\d+)$", "0000000000$1"); ciks[ix] = ReplaceInStringRegex(s1, "0*(\\d{10})$", "$1"); cccs[ix] = values[1]; passwords[ix] = values[2]; if (values[3] != "") { default_cik = ix; } }
We explode our file into lines using the ExplodeString SDK function. Leaving the delimiter undefined causes the function to use line endings as the default delimiter. We now have an array of CSV lines stored in our lines variables. We can then loop through that array and break apart each line, again with the ExplodeString function with a comma specified as a delimiter. Storing each property into our values string array, we then check to see if the first position of that array is empty, which would signify the end of our data. There are better methods of parsing CSV data that deal with quoted values but since the data in our table doesn’t need to be quoted this method is easier.
The format of the agents.csv file can been derived from reading the code but here is a sample file:
0000000001,accc4u$e,password,default 0000000002,2ccc4u$e,passw0rd,
A couple of regular expression replacements in the strings using the ReplaceInStringRegex SDK function format our CIK values properly by padding with zeros before trimming our string to ten digits. With a properly formatted CIK, we can store it, as well as the corresponding CCC and the password, into our specific arrays for each information type. Having completed parsing our CSV data, the load_agent_table function returns with no error.
Now back to our on_open_project function. We now need to retrieve the filer CIK from the project we’re opening. To do this, we call another programmer-defined function: read_cik.
dword window_type; handle edit_window; edit_window = GetEditWindowHandle(); if (IsError(edit_window)) { return ""; } window_type = GetEditWindowType(edit_window); window_type &= EDX_TYPE_ID_MASK; if (window_type == EDX_TYPE_EDGAR_VIEW) { return read_cik_offset(edit_window, "section_filer", 3, 1); } if (window_type == EDX_TYPE_XML_13F_VIEW) { return read_cik_offset(edit_window, "cik", 0, 0); } if (window_type == EDX_TYPE_XML_13H_VIEW) { return read_cik_offset(edit_window, "filerId", 0, 0); } // EDX_TYPE_XML_17_VIEW - Not defined in current SDK version if (window_type == 0x000001E0) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_17H_VIEW - Not defined in current SDK version if (window_type == 0x00000210) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_C_VIEW - Not defined in current SDK version if (window_type == 0x000001D0) { return read_cik_offset(edit_window, "cik", 0, 0); } if (window_type == EDX_TYPE_XML_D_VIEW) { return read_cik_formd(edit_window); } if (window_type == EDX_TYPE_XML_MA_VIEW) { return read_cik_offset(edit_window, "filerId", 0, 0); } if (window_type == EDX_TYPE_XML_MFP_VIEW) { return read_cik_mfp(edit_window); } // EDX_TYPE_XML_SDR_VIEW - Not defined in current SDK version if (window_type == 0x000001A8) { return read_cik_offset(edit_window, "cik", 0, 0); } // EDX_TYPE_XML_RGA_VIEW - Not defined in current SDK version if (window_type == 0x000001C0) { return read_cik_offset(edit_window, "cik", 0, 0); } return "";
This function first retrieves a handle to the edit window using the GetEditWindowHandle SDK function. With this handle, we can then retrieve the type of edit window using the GetEditWindowType function. We need to know this to appropriately locate the CIK. To get the specific type, we use the bitwise AND operator and combine the dword that was returned with the EDX_TYPE_ID_MASK. This will reveal the specific type of the window. With a switch/case structure, the function can examine the different window types and act accordingly. Depending on the type, we call an appropriate read_cik_offset function. Some of the constant values that are used here are not yet defined in the Legato SDK. If you think the SDK is missing any constant definitions, please contact Novaworks support.
Let’s look at an example:
// Read CIK using Offset string read_cik_offset(handle edit_window, string name, int offy, int offx) { string field; handle dataview; int location[2]; dataview = DataViewGetObject(edit_window, 0); field = DataViewFindCellByName(dataview, name); if (field != "") { location = CellAddressToIndex(field); return DataViewCellGetText(dataview, location[0] + offy, location[1] + offx); } return ""; }
These functions (read_cik_offset, read_cik_formd, and read_cik_mfp) take different arguments and operate slightly differently, but they all work to locate the CIK in the specific form type. They all do this through getting a handle to the data view using the DataViewGetObject SDK function (which requires the handle to the edit window, which we passed to the function). The DataViewFindCellByName function locates the position of the CIK for this form type. Then, with the CellAddressToIndex and DataViewCellGetText functions, we can retrieve the string containing the CIK. Note that the use of all these functions varies on form type and the exact position of the CIK within the data view object.
Once the position of the CIK is determined, these functions return a string containing the CIK. This, in turn, gets returned back to our on_open_project function. Now we can check our CSV information for a matching CIK. We do this with a while loop, which we leave if a match is found. If no match is found, we set the CIK to a default value.
// Get Filer CIK s1 = read_cik(); size = ArrayGetAxisDepth(ciks); ix = 0; while (ix < size) { if (s1 == ciks[ix]) { break; } ix++; } if (ix >= size) { ix = default_cik; }
If we find a match, we can then query the user about replacing the EDGAR Agent preferences with the corresponding information for that CIK (which is specific to a particular company). If the user selects yes, we can use the EDGARSetCredentials function to set that particular CIK, CCC, and password. Then we return without an error.
That describes the on_load_project function, which has to examine the CIK of an existing project with the CSV list, determine if there’s a match, and then load the information that’s relevant. The on_new_project function performs some similar tasks, though this time we don’t have a project CIK to which we can match information since the project is new. Therefore, we’re going to want to load some sort of default settings.
if (mode != "postprocess") { return ERROR_NONE; } // Check Type edit_window = GetEditWindowHandle(); if (IsError(edit_window)) { return ERROR_NONE; } window_type = GetEditWindowType(edit_window); window_type &= EDX_TYPE_ID_MASK; if ((window_type != EDX_TYPE_EDGAR_VIEW) && (window_type != EDX_TYPE_XML_13F_VIEW) && (window_type != EDX_TYPE_XML_13H_VIEW) && // EDX_TYPE_XML_17_VIEW - Not defined in current SDK version (window_type != 0x000001E0) && // EDX_TYPE_XML_17H_VIEW - Not defined in current SDK version (window_type != 0x00000210) && // EDX_TYPE_XML_C_VIEW - Not defined in current SDK version (window_type != 0x000001D0) && (window_type != EDX_TYPE_XML_D_VIEW) && (window_type != EDX_TYPE_XML_MA_VIEW) && (window_type != EDX_TYPE_XML_MFP_VIEW) && // EDX_TYPE_XML_SDR_VIEW - Not defined in current SDK version (window_type != 0x000001A8) && // EDX_TYPE_XML_RGA_VIEW - Not defined in current SDK version (window_type != 0x000001C0)) { return ERROR_NONE; } rc = load_agent_table(); if (IsError(rc)) { return rc; } // Set to Default rc = EDGARSetCredentials(ciks[default_cik], passwords[default_cik], cccs[default_cik]); if (IsError(rc)) { MessageBox('x', GetLastErrorMessage()); } return ERROR_NONE; }
Similar to the read_cik function, we retrieve the handle to the edit window and determine the window’s type. If the edit window doesn’t contain one of the EDGAR forms listed, we return with no error; there are no EDGAR preferences to set. We then call our load_agent_table function to load our CSV data. Let’s revisit that function briefly to examine something that wasn’t pertinent before:
if (values[3] != "") { default_cik = ix; }
This occurs inside the load_agent_table function inside our parsing loop. When we were dealing with loading a project that already has a CIK attached to it, the default CIK from our CSV file wasn’t relevant. Now, when beginning a new project, we’d like to set the EDGAR preferences to some default CIK, CCC, and password. This conditional statement is executed for every line of the CSV file, and it searches to find the line that is marked as default (with a non-empty value for the fourth column). We store the default CIK position.
Knowing the default CIK position, we can again use the EDGARSetCredentials function to set our CIK, CCC, and password from the CSV file.
// Set to Default rc = EDGARSetCredentials(ciks[default_cik], passwords[default_cik], cccs[default_cik]); if (IsError(rc)) { MessageBox('x', GetLastErrorMessage()); }
And that, as they say, is that. This script is fairly lengthy (approximately 300 lines of codes), but it performs a really useful function. It allows you to populate EDGAR preferences automatically by matching a CIK associated with the project being loaded to a list of EDGAR information and to load default filer information for a new project. This can save quite a bit of time, and it demonstrates how you can customize GoFiler to your specific needs. Next week, we’ll take a look at how to secure a script, and we’ll use this script as an example.
David Theis has been developing software for Windows operating systems for over fifteen years. He has a Bachelor of Sciences in Computer Science from the Rochester Institute of Technology and co-founded Novaworks in 2006. He is the Vice President of Development and is one of the primary developers of GoFiler, a financial reporting software package designed to create and file EDGAR XML, HTML, and XBRL documents to the U.S. Securities and Exchange Commission. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato