This week our blog will focus on another common task that can be automated with GoFiler. When working on an XBRL file, test filing it with the SEC is very routine. GoFiler can validate it, but there is no substitute for pushing it up to the SEC to see if it will pass or fail a test filing. Right now, to test file an XBRL file, a project must be created, and the XFR file must be exported to an XBRL fileset. Then the files should be attached to the project. Then you need to attach a dummy HTML or TXT file onto the project as a primary document before finally test filing it. This whole process can take 3-4 minutes . While that doesn’t sound like a lot, it can be annoying to have to do it over and over again while working on multiple filings. Using Legato, however, we can streamline this process down to a single button press.
Friday, May 26. 2017
LDC #36: Automated Test Filing XBRL, Part 1
To automate this, we will need to make template project files. We need templates so that we don’t have to ask for user input on the CIK, CCC, or any other filing information required to submit a test filing. If we try to test file a 10-Q, for example, we want it to open our 10-Q template project file, export our XBRL, attach it to our template, and test file it. These template files will need to be saved at a known location, be empty of attached files, and have a specific name so we can find the right template later. As these templates have so many requirements, writing a script to help create a template file is probably a good idea, and we will be exploring that today. Next week’s post will cover exporting the XFR file, attaching the exported contents to a copy of the project, and test filing the project.
Overall, this task will require three script files:
1) SaveTemplate.ms - This script will add our menu hooks to save an open project as an XBRL test template, and it will add a hook to open our templates directory in case we need to edit a template later.
2) TestFileXBRLDefinitions.ls - This file is very short and contains the location of our templates folder. By putting it in its own file, we can include it in both of our other scripts, so if we ever want to change the location of the templates folder it’s easier to edit it in only one place. It only contains definitions, and we do not want it to run on its own at application startup, so it has a .ls extension instead of a .ms extension.
3) TestFileXBRL.ms - This script will actually export the XBRL file and test file it. It will be covered next week.
The entire process, once completed with both scripts, would look like this:
1) User creates a new project file in GoFiler, as normal.
2) User fills out the project file with all required information but attaches no documents.
3) User goes to File->Tools, and selects “Save as XBRL Test Template” to create a template.
4) User can now open XFR files and test file them with one command by going to XBRL->Tools and selecting “Test File XBRL”.
An additional menu hook is also added in today’s script called “Open Templates Folder”, which simply opens the defined template folder so a user can edit existing templates easier. Let’s take a look inside TestFileXBRLDefinitions.ls:
#define TEMPLATE_FOLDER_NAME "XBRL Test Templates" #define TEMPLATE_FOLDER_LOCATION GetApplicationDataFolder()
This is a very basic file. It defines our template folder name and the template folder location. The default for the folder location is the Application Data folder, which every user should have read/write access to by default. Normal GoFiler templates are in the Program Files folder, but because that one often isn’t editable by normal users, we’ll use the appdata directory as a default instead.
Now let’s examine SaveTemplate.ms:
//
// GoFiler - Save Template
// ------------------------------------
//
// Saves an open project as a template for testing XBRL
//
// Revised 05-23-2017 SCH Page Created
//
// (c) 2017 Novaworks, LLC. All rights reserved.
//
#include "TestFileXBRLDefinitions.ls"
int setup_save();
int setup_open();
int setup();
int run(int f_id, string mode);
int open(int f_id, string mode);
int read_form_type();
boolean is_project_open();
/****************************************/
int setup(){ /* primary setup function */
/****************************************/
setup_save(); /* setup the save function */
setup_open(); /* setup the open function */
return ERROR_NONE; /* return */
} /* */
/****************************************/
int setup_save(){ /* setup function */
/****************************************/
string item[]; /* params of menu item */
string script; /* script running */
int rc; /* return code */
/* */
item["Code"]="XBRL_CREATE_TEMPLATE"; /* function name */
item["MenuText"]="Save as XBRL Test Template"; /* menu item */
item["Description"]="Save open project as a template "+ /* description */
"for XBRL Test Filing"; /* description */
MenuAddFunction(item); /* add item to the menu */
script = GetScriptFilename(); /* get name of this script */
MenuSetHook(item["Code"],script,"run"); /* set hook to run function */
return ERROR_NONE; /* return */
} /* */
/****************************************/
int setup_open(){ /* setup function */
/****************************************/
string item[]; /* params of menu item */
string script; /* script running */
int rc; /* return code */
/* */
item["Code"]="XBRL_OPEN_TEMPLATE"; /* function name */
item["MenuText"]="Open Templates Folder"; /* menu item */
item["Description"]="Open the folder containing test file "+ /* description */
"templates for XBRL Test Filing"; /* description */
MenuAddFunction(item); /* add item to the menu */
script = GetScriptFilename(); /* get name of this script */
MenuSetHook(item["Code"],script,"open"); /* set hook to run function */
return ERROR_NONE; /* return */
} /* */
/****************************************/
int run(int f_id, string mode){ /* run save function */
/****************************************/
dword wType; /* the type of edit window open */
string output; /* final location of output file */
string project_type; /* get the type of the project */
int rc; /* return code */
string template_folder; /* folder for templates */
handle hWindow; /* edit window handle */
/* */
if (mode!="preprocess"){ /* if not preprocess */
return ERROR_NONE; /* exit */
} /* */
if (is_project_open()==false){ /* if we do not have a project open */
MessageBox('x',GetLastErrorMessage()); /* display error message */
return ERROR_EXIT; /* exit with error */
} /* */
template_folder = AddPaths(TEMPLATE_FOLDER_LOCATION, /* build path to templates folder */
TEMPLATE_FOLDER_NAME); /* build path to templates folder */
if (IsFolder(template_folder)==false){ /* if the template folder doesn't exist */
rc = CreateFolder(template_folder); /* create it */
if (IsError(rc)){ /* if we cannot create it */
MessageBox('x',"Cannot create folder %s, error %0x", /* display error */
template_folder,rc); /* display error */
return ERROR_EXIT; /* return with error */
} /* */
} /* */
project_type = read_form_type(); /* read the form type */
rc = ProjectGetEntryCount(); /* make sure the project is empty */
if (rc>0){ /* if we had an error */
MessageBox('x',"Project must be empty to make a template."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
output = AddPaths(template_folder,project_type+".gfp"); /* save output */
rc = RunMenuFunction("FILE_SAVE_AS","Filename:"+output); /* save file */
if (IsError(rc)){ /* if it failed to save */
MessageBox('x',"Cannot save file, error %0x",rc); /* display error */
return ERROR_EXIT; /* return with error */
} /* */
MessageBox('i',"Template saved to %s",output); /* give user feedback */
return ERROR_NONE; /* return */
} /* */
/****************************************/
int open(int f_id, string mode){ /* open the template folder */
/****************************************/
string file; /* file to open */
/* */
if (mode!="preprocess"){ /* if not preprocess */
return ERROR_NONE; /* bail */
} /* */
file = BrowseOpenFile("Select Template File",".gfp files | *.gfp", /* */
AddPathDelimiter(AddPaths(TEMPLATE_FOLDER_LOCATION, /* browse for file to open */
TEMPLATE_FOLDER_NAME))); /* browse for file to open */
if (GetLastError()==ERROR_CANCEL){ /* if last error is error cancel */
return ERROR_NONE; /* return no error */
} /* */
RunMenuFunction("FILE_OPEN","Filename:"+file); /* open the file */
return ERROR_NONE; /* return no error */
} /* */
/****************************************/
string read_form_type(){ /* Get the type of the form */
/****************************************/
int type_index[2]; /* index of type */
string type_address; /* address of file type */
string type; /* type of form */
handle edit_window; /* edit window handle */
handle dataview; /* data view within the edit window */
/* */
edit_window = GetEditWindowHandle(); /* get handle to edit window */
dataview = DataViewGetObject(edit_window,0); /* get datasheet of project */
type_address = DataViewFindCellByName(dataview,"submissionType"); /* get address to submission type */
if(type_address==""){ /* if we cannot get the address */
MessageBox('x',"Cannot find submission type."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
type_index = CellAddressToIndex(type_address); /* convert address to index */
type = DataViewCellGetText(dataview,type_index[0],type_index[1]); /* get filing type */
if(type==""){ /* if we still dont' know type */
MessageBox('x',"Cannot find submission type."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
return type; /* return the type */
} /* */
/****************************************/
boolean is_project_open(){ /* tests if a project file is open */
/****************************************/
handle hWindow; /* window handle */
dword wType; /* type of window */
/* */
hWindow = GetActiveEditWindow(); /* get the active edit window */
if (IsError(hWindow)){ /* if we cannot get the active window */
SetLastError(ERROR_EXIT,"Cannot save as template, no file open.");/* set error message */
return false; /* return false */
} /* */
wType = GetEditWindowType(hWindow); /* get the type of the window */
if (IsError()){ /* if we cannot get the type of window */
hWindow = GetParentWindow(hWindow); /* get the parent of the window */
wType = GetEditWindowType(hWindow); /* get the type of the parent */
} /* */
wType &= EDX_TYPE_ID_MASK; /* mask form type */
if (wType != EDX_TYPE_PSG_PROJECT_VIEW && /* if not in project view */
wType != EDX_TYPE_EDGAR_VIEW){ /* if not a project file */
SetLastError(ERROR_EXIT,"Cannot save as template, "+ /* display error */
"no compatible project open."); /* display error */
return false; /* exit with error */
} /* */
return true; /* return no error */
}
/****************************************/
int main(){ /* main */
/****************************************/
setup(); /* run setup */
return ERROR_NONE; /* return */
}
The setup function we use this time is slightly different from others in the past. Instead of having a single setup function, there are two defined: setup_save and setup_open. Each of these adds and hooks to a different menu item. The main setup function, setup, is only responsible for calling each of these other two functions. The setup_save and setup_open functions work exactly as other setup functions have in previous scripts. They define an array of parameters for the menu item, use the AddMenuFunction SDK function to add it to the menu, and then use the MenuSetHook function to attach a specific function defined in this script to that menu item.
/****************************************/
int setup(){ /* primary setup function */
/****************************************/
setup_save(); /* setup the save function */
setup_open(); /* setup the open function */
return ERROR_NONE; /* return */
} /* */
/****************************************/
int setup_save(){ /* setup function */
/****************************************/
string item[]; /* params of menu item */
string script; /* script running */
int rc; /* return code */
/* */
item["Code"]="XBRL_CREATE_TEMPLATE"; /* function name */
item["MenuText"]="Save as XBRL Test Template"; /* menu item */
item["Description"]="Save open project as a template "+ /* discription */
"for XBRL Test Filing"; /* description */
MenuAddFunction(item); /* add item to the menu */
script = GetScriptFilename(); /* get name of this script */
MenuSetHook(item["Code"],script,"run"); /* set hook to run function */
} /* */
/****************************************/
int setup_open(){ /* setup function */
/****************************************/
string item[]; /* params of menu item */
string script; /* script running */
int rc; /* return code */
/* */
item["Code"]="XBRL_OPEN_TEMPLATE"; /* function name */
item["MenuText"]="Open Templates Folder"; /* menu item */
item["Description"]="Open the folder containing test file "+ /* discription */
"templates for XBRL Test Filing"; /* description */
MenuAddFunction(item); /* add item to the menu */
script = GetScriptFilename(); /* get name of this script */
MenuSetHook(item["Code"],script,"open"); /* set hook to run function */
return ERROR_NONE; /* return */
} /* */
The is_project_open function is responsible for testing if a compatible project is open. If so, it returns true. Otherwise, it just puts sets up an error message and returns false. To check the project, we need to first use the GetActiveEditWindow function to get the currently active edit window handle. If it cannot get a window handle, it means no file is open, so the function can set error message with the SetLastError function and return false. If we can get the window handle, we can then get the type using the GetEditWindowType function. Using the IsError function next, we can test if the last run function was an error. If the last function was an error, then we know we failed to get the type. The only reason that could happen is if we were in a sub-window (like Code View in an HTML file maybe), so we can use the GetParentWindow function to get the parent of our active window and then employ the GetEditWindowType function again. Once we have the type, we can get the type ID by using the bitwise AND operator it with the EDX_TYPE_ID_MASK define. Then we can test if our type is EDX_TYPE_PSG_PROJECT_VIEW or EDX_TYPE_EDGAR_VIEW, which would be two allowable project types. See the documentation of the GetEditWindowType function for more information on the predefined types. If the type is neither of these, we set an error and return false. Otherwise, we can simply return true.
/****************************************/
boolean is_project_open(){ /* tests if a project file is open */
/****************************************/
handle hWindow; /* window handle */
dword wType; /* type of window */
/* */
hWindow = GetActiveEditWindow(); /* get the active edit window */
if (IsError(hWindow)){ /* if we cannot get the active window */
SetLastError(ERROR_EXIT,"Cannot save as template, no file open.");/* set error message */
return false; /* return false */
} /* */
wType = GetEditWindowType(hWindow); /* get the type of the window */
if (IsError()){ /* if we cannot get the type of window */
hWindow = GetParentWindow(hWindow); /* get the parent of the window */
wType = GetEditWindowType(hWindow); /* get the type of the parent */
} /* */
wType &= EDX_TYPE_ID_MASK; /* mask form type */
if (wType != EDX_TYPE_PSG_PROJECT_VIEW && /* if not in project view */
wType != EDX_TYPE_EDGAR_VIEW){ /* if not a project file */
SetLastError(ERROR_EXIT,"Cannot save as template, "+ /* display error */
"no compatible project open."); /* display error */
return false; /* exit with error */
} /* */
return true; /* return no error */
}
The next function we’ll look at is read_form_type. It assumes we have a valid project file open from which we can read the form type, so it should only be used after is_project_open is called to test for valid open projects. First, read_form_type gets the edit window handle with the GetEditWindowHandle SDK function. Then, it retrieves a handle to the data view from that window with the DataViewGetObject function. The data view is an object that GoFiler uses to display things like project files or XBRL. You can access it directly if you get a handle to it, like we just did. We then want to find the address of the type by using the DataViewFindCellByName function to search for a data cell named “submissionType”, which is the name GoFiler uses by default for this particular data cell. The DataViewFindCellByName function always returns a string address. If it returns nothing, then we can display an error message and return. Otherwise, we can use the CellAddressToIndex function to convert our address to x and y coordinates for the cell’s location on our data sheet. Using the DataViewCellGetText function, we can pass in our view and our x and y coordinates and get back out the text of our cell. This should be the data type. If it’s blank, that means we cannot get the type, so we show an error and return an error. Otherwise, we’ll return the type.
/****************************************/
string read_form_type(){ /* Get the type of the form */
/****************************************/
int type_index[2]; /* index of type */
string type_address; /* address of file type */
string type; /* type of form */
handle edit_window; /* edit window handle */
handle dataview; /* data view within the edit window */
/* */
edit_window = GetEditWindowHandle(); /* get handle to edit window */
dataview = DataViewGetObject(edit_window,0); /* get datasheet of project */
type_address = DataViewFindCellByName(dataview,"submissionType"); /* get address to submission type */
if(type_address==""){ /* if we cannot get the address */
MessageBox('x',"Cannot find submission type."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
type_index = CellAddressToIndex(type_address); /* convert address to index */
type = DataViewCellGetText(dataview,type_index[0],type_index[1]); /* get filing type */
if(type==""){ /* if we still dont' know type */
MessageBox('x',"Cannot find submission type."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
return type; /* return the type */
} /* */
Our run function is what’s actually hooked into the menu by our setup function, and it calls the two functions above to help make our template file. First, it tests to ensure we’re running in preprocess mode. If not, we can exit. Then it checks if a project file is open with our is_project_open function. If there’s no project open, it displays the error message set by the is_project_open function with the GetLastErrorMessage and MessageBox functions and exits. Otherwise, we keep going. Next the run function builds our template folder path by using the AddPaths function on our defines, and it tests if this path is already a folder with the IsFolder function. If there’s no folder at that location, it runs the CreateFolder function to make one. The IsError function is used to determine if it succeeded. If the function failed, we display an error and return.
Now that we have our folder and we know a project is open, we can read the project type with the read_form_type function. We also use the ProjectGetEntryCount function to get the number of entries attached to a project. This is going to be a template, so if there are any entries at all we should display an error message and return. If there are no entries, we can keep going and use the AddPaths function again to build a final output path. Then the RunMenuFunction triggers Save As, so we can save this file to our templates folder. We need to check the return code from the Save operation to ensure it worked and display a failure message if it didn’t or a success message if it did.
/****************************************/
int run(int f_id, string mode){ /* run save function */
/****************************************/
dword wType; /* the type of edit window open */
string output; /* final location of output file */
string project_type; /* get the type of the project */
int rc; /* return code */
string template_folder; /* folder for templates */
handle hWindow; /* edit window handle */
/* */
if (mode!="preprocess"){ /* if not preprocess */
return ERROR_NONE; /* exit */
} /* */
if (is_project_open()==false){ /* if we do not have a project open */
MessageBox('x',GetLastErrorMessage()); /* display error message */
return ERROR_EXIT; /* exit with error */
} /* */
template_folder = AddPaths(TEMPLATE_FOLDER_LOCATION, /* build path to templates folder */
TEMPLATE_FOLDER_NAME); /* build path to templates folder */
if (IsFolder(template_folder)==false){ /* if the template folder doesn't exist */
rc = CreateFolder(template_folder); /* create it */
if (IsError(rc)){ /* if we cannot create it */
MessageBox('x',"Cannot create folder %s, error %0x", /* display error */
template_folder,rc); /* display error */
return ERROR_EXIT; /* return with error */
} /* */
} /* */
project_type = read_form_type(); /* read the form type */
rc = ProjectGetEntryCount(); /* make sure the project is empty */
if (rc>0){ /* if we had an error */
MessageBox('x',"Project must be empty to make a template."); /* display error */
return ERROR_EXIT; /* return error */
} /* */
output = AddPaths(template_folder,project_type+".gfp"); /* save output */
rc = RunMenuFunction("FILE_SAVE_AS","Filename:"+output); /* save file */
if (IsError(rc)){ /* if it failed to save */
MessageBox('x',"Cannot save file, error %0x",rc); /* display error */
return ERROR_EXIT; /* return with error */
} /* */
MessageBox('i',"Template saved to %s",output); /* give user feedback */
return ERROR_NONE; /* return */
} /* */
The last function is the open function. This one is really basic, we just think it’s a minor, quality of life improvement. It opens a File Open dialog to the templates directory, so a user can edit a template file quickly without having to remember where the templates folder is. It starts by ensuring it’s running in preprocess mode. Then the function runs the BrowseOpenFile function to query the user to pick a file to open in the templates folder. If the user cancels the operation, it returns. Otherwise, it runs the RunMenuFunction function to trigger a File Open operation on the selected file, so it can be edited.
/****************************************/
int open(int f_id, string mode){ /* open the template folder */
/****************************************/
string file; /* file to open */
/* */
if (mode!="preprocess"){ /* if not preprocess */
return ERROR_NONE; /* bail */
} /* */
file = BrowseOpenFile("Select Template File",".gfp files | *.gfp", /* */
AddPathDelimiter(AddPaths(TEMPLATE_FOLDER_LOCATION, /* browse for file to open */
TEMPLATE_FOLDER_NAME))); /* browse for file to open */
if (GetLastError()==ERROR_CANCEL){ /* if last error is error cancel */
return ERROR_NONE; /* return no error */
} /* */
RunMenuFunction("FILE_OPEN","Filename:"+file); /* open the file */
return ERROR_NONE; /* return no error */
} /* */
Using this script, we can now easily create and open our filing templates. These templates will be used in our next script to test file XBRL documents quickly and easily. The script next week will handle the actual test filing process.
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
